Skip to content

fix(scripts): check.sh says what it covered, on stdout (#336) - #372

Merged
JArmandoAnaya merged 1 commit into
mainfrom
fix/336-check-verdict
Aug 5, 2026
Merged

fix(scripts): check.sh says what it covered, on stdout (#336)#372
JArmandoAnaya merged 1 commit into
mainfrom
fix/336-check-verdict

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #336.

The issue's first criterion was already met by PR #249 — the script aborts on a missing node_modules rather than limping on. What survived is that it says so on stderr, leaving stdout empty, so a caller capturing stdout sees a partial run and a full one as the same thing: some green pytest output, then silence. The exit code is right, and nobody reads an exit code out of a transcript. It is the same false-calm failure the script's own header warns about for | tail, arriving from the other direction.

What changed

One last line on stdout, from a trap … EXIT — so a require_node_modules three groups deep cannot skip it, which is the exact run that most needed to say something:

check.sh: PASSED  ran=python,frontend,generated,browser  skipped=none
check.sh: PASSED  ran=frontend,generated  skipped=python,browser
check.sh: INCOMPLETE  ran=none  skipped=python,frontend,generated,browser
  • ran= is what completed, never what was asked for — recorded after each group's function returns, so a group that aborted partway through is never counted as covered. skipped= is every group the script knows that is not in it.
  • Three outcomes, because two would blur the news. FAILED is a step reporting a problem; INCOMPLETE is the run leaving early, where nothing was found wrong with the tree and the checks simply did not happen.
  • The browser banner now follows what ran, not what was requested — which is what its own comment always claimed ("a partial run is the same lie however it was asked for"), and was not true of a run that asked for everything and died in frontend. It stays quiet when nothing ran, so unknown group 'nope' is not buried under twelve lines about Playwright.
  • Ordering is unchanged in spirit: the banner still comes after the verdict it qualifies — now after the machine-readable line as well, since that is the line it is qualifying.

Written without arrays for the accumulator (ran is a comma string): macOS ships bash 3.2, where an empty array under set -u is an unbound variable. That constraint is already documented in the timing block; this way there is nothing to remember.

Found, not fixed

Nothing.

Test plan

New tests/scripts/check_stages.test.mjs — 8 cases, ~0.2s, no new dependency. Two halves that fail differently:

  • Behavioural — copies the real script into a temp root with nothing in it (the issue's own reproduction) and runs it. generated → exit 2 with the verdict line on stdout alone; pythonFAILED ran=python; an unknown group → INCOMPLETE ran=none; the banner quiet with nothing run and loud when browser was requested but never reached. The subprocess gets a PATH without uv or pnpm, so every step fails identically on every machine rather than depending on what happens to be installed.
  • Static — the group roster, the dispatch case, the default list, --fast, and the trap. This half exists because the way the script rots is not a deleted line but a group quietly leaving the loop: coverage shrinks, every remaining stage passes, and the verdict says PASSED about a smaller run than the reader believes.

Mutation checks, anchors asserted before and after each patch

mutation caught by
trap summary EXIT commented out 6 of 8
ran never recorded 3
browser dropped from ALL_GROUPS 5
browser) run_browser ;; deleted from the dispatch 1 — the static roster test, and nothing else
the default run drops generated 1

The fourth is the one the issue is really about: every behavioural test stays green while the script silently stops running a quarter of the gate. That is why the static half is there.

Full gate, staged against the harness's ~10-minute ceiling

Test directories derived from ls tests/ at run time.

stage exit
pytest tests/architecture tests/cli tests/examples tests/formats tests/jobs tests/packaging tests/scripts tests/test_versioning.py 0 (522)
pytest tests/kernel tests/mcp tests/server 0 (2080)
ruff check . / ruff format --check . / mypy src/visionset / lint-imports 0 / 0 / 0 / 0
check.sh frontend generated 0 — and it ended check.sh: PASSED ran=frontend,generated skipped=python,browser
CI=1 npx playwright test 0 (197 passed, 3.0m)
CI=1 npx playwright test -c playwright.cycle.config.ts 0 (1 passed, 36.7s)

The script aborted correctly on a missing prerequisite and said so on
stderr, leaving stdout empty — so a caller capturing stdout could not
tell a partial run from a full one. The exit code was right, and nobody
reads an exit code out of a transcript.

Every run now ends with one machine-readable line on stdout naming what
completed, what did not, and the verdict, printed from a trap so no exit
path can skip it. INCOMPLETE is kept apart from FAILED: the checks not
happening and the checks finding something are different news.

The browser banner follows what ran rather than what was requested, which
is what its own comment always claimed, and stays quiet when nothing ran
at all. tests/scripts/check_stages.test.mjs guards the line and holds the
group roster to the dispatch, since the way this rots is a stage quietly
leaving the loop while everything left still passes.
@JArmandoAnaya
JArmandoAnaya enabled auto-merge (squash) August 5, 2026 21:11
@JArmandoAnaya
JArmandoAnaya merged commit 1cca0c9 into main Aug 5, 2026
14 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the fix/336-check-verdict branch August 5, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check.sh: a partial run is indistinguishable from a full one on stdout, and no test guards it

1 participant