tests: derive block tallies from their own output (fixes #654)#661
Merged
Conversation
…ynced literals (#654) 23 suite blocks hard-coded a "(N checks)" label plus a `TOTAL/PASS += N` literal that had to be kept in sync with the test file by hand. The literals had drifted ~169 asserts below reality (headline 2890, true 3059), and the same hand-sync could just as easily over-report — a real coverage lie. Derive the count from each block's own output instead: - `derive_count()` parses `Tests: N` from the block output — the lib/test.eigs harness line and the hand-rolled `<Name> Tests: N/M passed` equivalents. $2 is the documented fallback (the historical literal), used when the output carries no count line (bare-marker suites, or a crashed/timed-out run). An UNEXPECTED fallback (a block that normally self-reports but this run did not) is surfaced with a visible NOTE on stderr, so it is never silent. - 27 hand-rolled blocks whose files emit a count line now tally the derived count and drop the stale "(N checks)" parenthetical; the true count surfaces in the post-run PASS line. - check_eigs_suite derives from output when its declared count > 1 (a declared 1 is a deliberate single-gate and keeps 1). Its rc=124 timeout branch fails by the derived count, which for a no-count runaway is the declared fallback — same magnitude as before. Blocks whose output has no count line (e.g. test_string_math, test_file_io, audio/http/db) keep their literal as the documented fallback. Full suite green at 3059/3059 (PASS==TOTAL). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #657 by @Nitjsefnie — same commit, rebased onto current main. They authored this (git author preserved); I took over the rebase since their PR is from a fork I can't push to. Original review and verification: #657.
Rebase needed because #659 and #658 landed after their base. The one hunk that mattered was
[60] Terminal Builtins: their diff carries the old bare invocation as context, and #659 rewrote that exact line to add the stdin guard. Git's 3-way merge reconciled them onto different lines, keeping both #659's$EIGS_TMO ... </dev/nullguard and theirderive_countline — but that is precisely the case where a clean auto-merge can silently drop a fix, so I verified rather than trusted it.Verified on the rebased tree
NOTE(#654)fallbacks. The number is 3060 + 7 (W020's checks from lint: W020 flags unobserved: blocks that provably do nothing; fix README's example (#655) #658) — picked up automatically with zero runner edits, which is the whole point of this change.[60]before fix(runner): guard the test_terminal.eigs invocation (fixes #656) #659): completed rc=0, 3067/3067, in 143s. A dropped guard would have wedged there.What it does (from #657)
derive_count()parsesTests: Nfrom each block's own output, with the historical literal as a documented fallback; an unexpected fallback prints a visibleNOTE(#654)rather than degrading silently. Kills the hand-synced-literal drift class — spot-checked[35/36] Regex, which declared 15 while running 34.Credit to @Nitjsefnie for the design and implementation. Closes #654; supersedes #657.
🤖 Rebased and verified with Claude Code