Report unexercised semantic signals as SKIP, not PASS - #119
Merged
b-macker merged 1 commit intoAug 3, 2026
Conversation
test_semantic_signals.sh scored five assertions as PASS when the agent send failed. Group B is gated on GK1 being SET, not on it being valid, so a stale or revoked key reaches these branches: the suite reported the semantic_stability signal green on a run that never got past authentication. B03 was the clearest case — its message read "Semantic section check skipped" while calling pass(). C01-C03 are the older instance and the source of the pattern; C01 tests for a printed ERROR= before it reads SSC at all, so ANY caught exception scored as a satisfied assertion about a signal that never ran. "The signal did not fire" and "the signal never ran" are the same observation here, and only one of them is a result. skip() already exists in this file and counts toward TOTAL, so nothing is hidden. Reproduced with GK1 set to an invalid key: before, 20 PASS / 0 SKIP with B02, B03, C01, C02 and C03 all green on authentication failures; after, 15 PASS / 5 SKIP. The no-key path is unchanged (9 PASS / 11 SKIP) and a valid-key run is unaffected — only the error branches moved. 441 tests / 0 unexpected, 874 leak checks / 0 failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
b-macker
marked this pull request as ready for review
August 3, 2026 04:12
4 tasks
b-macker
pushed a commit
that referenced
this pull request
Aug 3, 2026
Three consecutive CI runs failed in a stub-backed suite, each time a DIFFERENT one, none reproducible locally: #119 Build & Test test_quarantine_corroboration.sh #120 build-linux test_failure_mode_coverage.sh #120 Build & Test test_absorption_degenerate.sh A code regression fails the same test every run. A different test each time, all sharing one start_stub idiom copied across 9 files, is the launcher. Two defects in it, both of which bite only under load: - the port is picked at random with no bind check and no retry, so a collision or a lingering TIME_WAIT socket leaves the stub dead - the readiness wait was a flat 5s, and python3 startup + bind exceeds that often enough to matter on a busy runner Either way start_stub returned 1, the caller printed STUB_FAIL, and every assertion in the suite then failed for a reason unrelated to what the test measures — with nothing in the output saying so. Now retries across 3 ports, waits 30s per attempt, and prints the stub log tail naming the real cause. The kill -0 check is what keeps the longer bound cheap: a stub that died on bind is detected immediately and retried on a fresh port instead of waiting out the ceiling. Verified the retry path by making every bind fail: 3 attempts, the actual error surfaced ("Address already in use"), rc=1 — in 0 seconds, not 90. The old code burned 5s silently and said nothing. All 9 patched suites pass individually (93 assertions), 441 tests / 0 unexpected, 874 leak checks / 0 failures. Not claimed: that this is the whole cause. It is the mechanism that fits every observation, and the next failure now names itself either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_semantic_signals.shscored five assertions as PASS when the agent send failed. Group B is gated onGK1being set, not on it being valid, so a stale or revoked key reaches these branches — and the suite reported thesemantic_stabilitysignal green on a run that never got past authentication.B03 was the clearest case: its message read "Semantic section check skipped" while calling
pass().Reproduction
GK1set to an invalid key, againstmaster:Nothing about signal 10 or signal 11 was measured. The suite was green.
C01–C03 are the older instance
B02/B03 arrived in #118; C01–C03 predate it and are the pattern that was copied. C01 is the worst of them — it tests for a printed
ERROR=before it readsSSCat all, so any caught exception scores as a satisfied assertion about a signal that never ran:Fixing only the new pair would have left the source in place, so all five move together.
The distinction
"The signal did not fire" and "the signal never ran" are the same observation here, and only one of them is a result. This is the same shape as
H01,L24-02,L25-03, and the two defects in my own work earlier in this campaign: an assertion satisfiable without the property holding, because absence of the bad outcome and absence of the mechanism look identical.skip()already exists in this file and incrementsTOTAL, so nothing is hidden — the count stays at 20 and the skips are visible in the summary line.Test Plan
bash run-all-tests.shwith no new failures — 441 tests, 0 unexpected failuresbash tests/security/test_error_msg_leaks.sh— 874 checks, 0 failuresDegraded-case check (invalid
GK1, the condition that produced the vacuous passes):Paths confirmed unchanged:
master(Group B was already fully skipped whenGK1is unset)Related Issues
Follow-up to #118. The Termux work that surfaced this was correct in substance —
test_consequence_proof.shandtest_d1_reconciliation.shboth usedskipon their API-error paths, which is the right three-way form.test_semantic_signals.shusedpass, matching a C-group pattern that was already wrong.Generated by Claude Code