Skip to content

Report unexercised semantic signals as SKIP, not PASS - #119

Merged
b-macker merged 1 commit into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Aug 3, 2026
Merged

Report unexercised semantic signals as SKIP, not PASS#119
b-macker merged 1 commit into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

@b-macker b-macker commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

test_semantic_signals.sh scored five assertions as PASS when the agent send failed. Group B is gated on GK1 being set, not on it being valid, so a stale or revoked key reaches these branches — and the suite reported the semantic_stability signal green on a run that never got past authentication.

B03 was the clearest case: its message read "Semantic section check skipped" while calling pass().

Reproduction

GK1 set to an invalid key, against master:

PASS [B02] On-topic test ran (API error, non-deterministic)
PASS [B03] Semantic section check skipped (API error in B02)
PASS [C01] On-topic pair test ran (API error, non-deterministic)
PASS [C02] Off-topic shift test ran (API error, non-deterministic)
PASS [C03] Mandate alignment test ran (API error, non-deterministic)

PASS: 20  FAIL: 0  SKIP: 0  TOTAL: 20

Nothing about signal 10 or signal 11 was measured. The suite was green.

C01–C03 are the older instance

B02/B03 arrived in #118; C01–C03 predate it and are the pattern that was copied. C01 is the worst of them — it tests for a printed ERROR= before it reads SSC at all, so any caught exception scores as a satisfied assertion about a signal that never ran:

if echo "$OUTPUT" | grep -q "ERROR="; then
    pass "C01" "On-topic pair test ran (API error, non-deterministic)"
elif [ "${SSC:-}" = "0" ]; then
    ...

Fixing only the new pair would have left the source in place, so all five move together.

The distinction

"The signal did not fire" and "the signal never ran" are the same observation here, and only one of them is a result. This is the same shape as H01, L24-02, L25-03, and the two defects in my own work earlier in this campaign: an assertion satisfiable without the property holding, because absence of the bad outcome and absence of the mechanism look identical.

skip() already exists in this file and increments TOTAL, so nothing is hidden — the count stays at 20 and the skips are visible in the summary line.

Test Plan

  • Ran bash run-all-tests.sh with no new failures — 441 tests, 0 unexpected failures
  • bash tests/security/test_error_msg_leaks.sh — 874 checks, 0 failures
  • Added/updated tests for new functionality — n/a, this corrects existing assertions
  • Tested manually in the REPL — n/a

Degraded-case check (invalid GK1, the condition that produced the vacuous passes):

before after
B02, B03, C01, C02, C03 PASS SKIP
totals 20 PASS / 0 SKIP 15 PASS / 5 SKIP

Paths confirmed unchanged:

  • No key at all — 9 PASS / 11 SKIP, identical to master (Group B was already fully skipped when GK1 is unset)
  • Valid key — untouched; only the error branches moved, every success branch is byte-identical

Related Issues

Follow-up to #118. The Termux work that surfaced this was correct in substance — test_consequence_proof.sh and test_d1_reconciliation.sh both used skip on their API-error paths, which is the right three-way form. test_semantic_signals.sh used pass, matching a C-group pattern that was already wrong.


Generated by Claude Code

test_semantic_signals.sh scored five assertions as PASS when the agent send
failed. Group B is gated on GK1 being SET, not on it being valid, so a stale
or revoked key reaches these branches: the suite reported the
semantic_stability signal green on a run that never got past authentication.
B03 was the clearest case — its message read "Semantic section check
skipped" while calling pass().

C01-C03 are the older instance and the source of the pattern; C01 tests for
a printed ERROR= before it reads SSC at all, so ANY caught exception scored
as a satisfied assertion about a signal that never ran.

"The signal did not fire" and "the signal never ran" are the same
observation here, and only one of them is a result. skip() already exists in
this file and counts toward TOTAL, so nothing is hidden.

Reproduced with GK1 set to an invalid key: before, 20 PASS / 0 SKIP with
B02, B03, C01, C02 and C03 all green on authentication failures; after,
15 PASS / 5 SKIP. The no-key path is unchanged (9 PASS / 11 SKIP) and a
valid-key run is unaffected — only the error branches moved.

441 tests / 0 unexpected, 874 leak checks / 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

@b-macker
b-macker marked this pull request as ready for review August 3, 2026 04:12
@b-macker
b-macker merged commit bf0c67e into master Aug 3, 2026
24 of 25 checks passed
@b-macker
b-macker deleted the claude/naab-inadmissible-action-prevention-4cmn1m branch August 3, 2026 04:12
b-macker pushed a commit that referenced this pull request Aug 3, 2026
Three consecutive CI runs failed in a stub-backed suite, each time a
DIFFERENT one, none reproducible locally:

  #119  Build & Test  test_quarantine_corroboration.sh
  #120  build-linux   test_failure_mode_coverage.sh
  #120  Build & Test  test_absorption_degenerate.sh

A code regression fails the same test every run. A different test each time,
all sharing one start_stub idiom copied across 9 files, is the launcher.

Two defects in it, both of which bite only under load:

  - the port is picked at random with no bind check and no retry, so a
    collision or a lingering TIME_WAIT socket leaves the stub dead
  - the readiness wait was a flat 5s, and python3 startup + bind exceeds
    that often enough to matter on a busy runner

Either way start_stub returned 1, the caller printed STUB_FAIL, and every
assertion in the suite then failed for a reason unrelated to what the test
measures — with nothing in the output saying so.

Now retries across 3 ports, waits 30s per attempt, and prints the stub log
tail naming the real cause. The kill -0 check is what keeps the longer bound
cheap: a stub that died on bind is detected immediately and retried on a
fresh port instead of waiting out the ceiling.

Verified the retry path by making every bind fail: 3 attempts, the actual
error surfaced ("Address already in use"), rc=1 — in 0 seconds, not 90. The
old code burned 5s silently and said nothing.

All 9 patched suites pass individually (93 assertions), 441 tests /
0 unexpected, 874 leak checks / 0 failures.

Not claimed: that this is the whole cause. It is the mechanism that fits
every observation, and the next failure now names itself either way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants