test(mcp): report what was counted, not just the failure count - #6
Merged
Conversation
Follow-up to #5, from review feedback on that PR. The summary line read `all _run contract tests passed (0 failed)`. A run that collected ZERO checks prints exactly the same thing, so the line could not tell "everything passed" apart from "nothing ran". The real evidence was the eight `ok` lines above it — but the summary is what a human quotes later, and it was the weakest statement in the file. This is the same rule the rest of this work applies elsewhere: a green without a denominator is not a measurement. `substance_check.py` was already reporting `n/n (...)`; the two summary lines in this repo now agree with each other. - summary is now `N/M checks passed`, so a sabotaged run reads `4/8` instead of a bare count - an empty run is an explicit failure rather than a vacuous pass, and it is decided BEFORE printing — a summary saying "passed" above a failure line is the same trap one line lower Also drops an em-dash from the MCP import step's output: it renders as `?` on the Windows console, which reads like a mojibake defect in the log when nothing is wrong. The test passes either way; this only stops the log from carrying a false signal. Verified in all three directions: normal run 8/8, fix reverted 4/8, no checks collected fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #5, acting on review feedback left on that PR.
The problem
The contract-test summary read:
A run that collected zero checks prints the same line. So it could not distinguish
"everything passed" from "nothing ran". The real evidence was the eight
oklines above it —but the summary is what gets quoted later, and it was the weakest statement in the file.
This is the same rule the rest of this work applies elsewhere: a green without a denominator
is not a measurement.
substance_check.pywas already reportingn/n (...), so the twosummary lines in this repo disagreed with each other. Now they don't.
What changed
N/M checks passed. A sabotaged run now reads4/8— the number says whichhalf broke, instead of just that something did.
saying "passed" above a failure line is the same trap one line further down.
?on the Windowsconsole, which reads like a mojibake defect in the log when nothing is actually wrong.
Cosmetic, but a log that carries a false signal costs someone a real investigation.
Verified in three directions
all _run contract tests passed: 8/8 checks passed_runfix revertedCONTRACT TESTS FAILED: 4/8 checks passedCONTRACT TESTS FAILED: no checks ran at all(exit 1)Scope: reporting only. No change to what is asserted or to
_runitself.