test: an involuntary skip is a failure — tools/_skip_policy.py and the rig lane (#178a)

[JC-000]

Part A of #178. Codifies the project's involuntary-skip rule as a callable and wires the rig lane to it.

Fourth revision. Three adversarial review rounds. Round 1 found five defects (all reproduced, all fixed in 8cc2d1c); round 2 verified those five and found one more — a whole-PR claim in this body that was false by omission — fixed in 76621b6; round 3 verified everything and found one stale comment, fixed in 4e90ffc. Corrected claims are rewritten below rather than quietly patched.

What it does

Adds tools/_skip_policy.py — three deliberately different calls, because the remedy decides the verdict:

call	verdict	exit	remedy
not_applicable()	voluntary — another rig owns the coverage	0, but as a named verdict, never a bare return 0	"use the other rig"
cannot_run()	involuntary — this IS the target config and it still could not run	2, distinct from 1 ("a check ran and failed")	"install something"
cannot_run(opt_out_env=None)	the things no env var may silence: a failed build, and contention	2, no hatch	—

Wired into eight entry points: the four tests/rig_phase*.py bridge rigs, tests/rig_vice_https_macos.py, tools/test_p384_symbols.py, tools/test_ecdsa_p384_kat.py, and tools/https_e2e/ (which gains a platform_supported() predicate so check_prerequisites()'s contract is unchanged).

Two new pure-logic pytest modules, both in pytest.ini testpaths: tools/test_skip_policy.py (19 cases, the helper) and tools/test_rig_skip_contract.py (12 cases, the call sites — a helper with no call-site test is just another convention to miss, which is #178's own thesis).

Why

Measured before the change, with build/ absent:

• tests/rig_phase{1_dhcp,2_http,3_https,3_https_1mhz}.py — exit 0 at the missing-prerequisite site and at return _skip("c64-https.prg could not be built"). A failed make laundered into a pass.
• tests/rig_vice_https_macos.py — exit 0 with the rig down; and SystemExit("build failed") exited 1, making a broken build indistinguishable from a real handshake failure.
• tools/test_p384_symbols.py — exit 0 on BACKEND != uci (voluntary, but silent).
• tools/test_ecdsa_p384_kat.py --u64 with no U64_HOST — printed U64: 0/0 passed and OVERALL: PASS, exit 0.

Behaviour changes, both directions, stated rather than discovered:

• Tightening. --u64 without U64_HOST is now refused up front, before the multi-hour VICE lane, and fails in ~0.1 s. It is not opt-out-able — see F1 in round 2 for why that matters.
• No loosening. Two gates in the first push did get weaker than master, in opposite corners of the tree; both are reverted (defect D3 in round 1, F1 in round 2). With those closed, no exit code in this PR is weaker than master's — and that claim has now been swept exhaustively by the reviewer across all seven wired files, master against branch, every return / SystemExit / sys.exit including every opt-out path.

Platform is asked FIRST and is a voluntary skip: the four bridge rigs are Linux-only, and on Darwin check_prerequisites() reports "ip not on PATH; iptables not on PATH", which is not installable there. Routing that to cannot_run() would make four rigs a permanent red on the project's primary platform, with the global opt-out as the only remedy — i.e. the policy gets disabled everywhere.

The opt-out requires the literal "1". Bare truthiness meant C64_ALLOW_SKIP=0 enabled it, so setting 0 to close the hatch disabled the whole policy. Every other gate in this repo (C64_SKIP_BUILD, VICE_HTTPS_OK_TO_RUN, …) compares to "1" and this one now does too.

Also narrowed the macOS rig's contention detector: pgrep -fl "ethernetioif feth0" matched the full command line of every process against an unanchored pattern (a grep, an editor, a driver script carrying that text all matched — demonstrated with a plain python3 -c). Now pgrep -x x64sc plus an argv check.

Review round 1 — five defects, all reproduced, all fixed in 8cc2d1c

D1. require()'s opt-out crashed outside pytest — and the hand-rolled copy it supersedes had already fixed this. import pytest succeeds whenever pytest is installed, never mind whether it is driving, and pytest.outcomes.Skipped derives from BaseException, which neither runner's except Exception can catch. With the sibling harness blocked:

$ C64_ALLOW_SKIP=1 python3 tools/test_rig_skip_contract.py
  ...
  File "tools/_skip_policy.py", line 279, in require
    pytest.skip(f"{text} [{opt_out_env}=1 set]")
Skipped: COULD NOT RUN: tools/https_e2e could not be imported ...
EXIT=1

No summary line, and the four source-inspection tests that could have run never ran. Exit 1 means "a check ran and failed" under the module's own contract, so it was wrong in both directions — and only on the escape hatch this PR advertises. Without the opt-out the same run was correct (4/10 passed, six named failures), which is why nothing caught it.

tools/test_uci_data_acc.py:631 had already solved this, with sys.modules.get("pytest") and a comment saying why. A shared helper that is worse than the copy it supersedes is not a refactor. Fixed by mirroring it: outside pytest, require() raises VoluntarySkip, a plain Exception. Either way the body does not run, so fail-closed is unchanged.

D2. The literal-"1" guard was skip-swallowed by the mutation it targets. test_require_still_raises_when_the_opt_out_is_zero was except SkipPolicyError: pass — vacuous against its own defect. Under bare truthiness, bool("0") is true, require() takes the opt-out branch, pytest records SKIPPED, exit 0. Measured: 1 skipped, not a failure. The guard turned itself into a green skip under precisely the defect it guards — the vacuous-skip shape #178 exists to kill, inside #178's own implementation — and it read as caught only because three sibling tests on the cannot_run() lane happen to fail. The catch is now BaseException-wide; under the same mutant the module reports 4 failed, 13 passed.

D3. Undisclosed loosening: VICE_HTTPS_OK_TO_RUN unset went 2 → 0. Reverted to exit 2, non-opt-out-able. Unset is the default state — it cannot tell a considered decline from a forgotten flag — and what the flag asserts ("the UCI HTTPS listener has stopped") is a contention claim, the one category this policy gives no opt-out at all; the port-443 check three lines below is the same interlock measured directly and was already exit 2. The only thing #178 now changes about this gate is that the verdict is a named block instead of a bare print.

D4. tests/rig_phase2_http.py contradicted itself inside forty lines. Its docstring said the macOS rig "owns the coverage"; its own _COUNTERPART string — the one that reaches the operator — says that rig drives TLS, "so plaintext HTTP specifically has no macOS rig". Load-bearing, because the voluntary-skip verdict is justified by "nothing is lost", which for this one rig is false. Exit 0 stays (no remedy on macOS), but the reason is now the true one and a new test pins the two halves together.

D5. Both _standalone() runners collapsed "could not run" onto 1. They now separate VoluntarySkip / SkipPolicyError / Exception and return 0 / 1 / 2. Measured on tools/test_rig_skip_contract.py with the harness blocked: exit 2 plain, exit 0 with C64_ALLOW_SKIP=1 (5/11 passed, 6 skipped by explicit opt-out), exit 0 with the harness present.

Red-then-green evidence

Every mutant below was run on this branch at dc06095, macOS, pytest 9.0.3.

M0′ — revert only the four bridge rigs, keep https_e2e. The sharp version of the contract mutant:

FAILED test_bridge_rigs_ask_platform_before_prerequisites
FAILED test_bridge_rigs_still_fail_when_the_platform_is_right
FAILED test_bridge_rigs_route_a_broken_build_to_two_without_an_opt_out
FAILED test_the_interlock_flag_unset_is_contention_and_stays_exit_two
FAILED test_phase2_does_not_claim_a_counterpart_it_does_not_have
5 failed, 6 passed

These are real assertions about exit codes, e.g.

E   AssertionError: rig_phase1_dhcp: broken build must be 2 even opted out, got 0
E     SKIP: c64-https.prg could not be built
E   assert 0 == 2

M1 — the "1" comparison. Replace _opted_out's .strip() == "1" with bool(...): 4 failed, 13 passed, including test_require_still_raises_when_the_opt_out_is_zero (which skipped before D2 was fixed).

D1 mutant — restore import pytest in require() → test_require_does_not_hand_a_skip_to_a_non_pytest_runner FAILS.
D3 mutant — re-route the interlock to not_applicable() → test_the_interlock_flag_unset_is_contention_and_stays_exit_two FAILS.
D4 mutant — restore the contradictory docstring → test_phase2_does_not_claim_a_counterpart_it_does_not_have FAILS.

Each restored → green.

Green, whole suite. Bare pytest at the repo root, with a BACKEND=uci USE_NISTCURVES_ONCHIP=1 PRG in build/ (sha256 5f9e9fb9e70edd2a15ab2249120f161d1826262140fa6ce6f3e01410a876b474):

86 passed in 7.27s

31 of those 86 are new (origin/master is 55 passed in the same conditions).

Review round 2 — one more defect, fixed in 76621b6

Round 2 re-verified all five round-1 fixes by reproducing the original scenarios (not by reading the diff) and confirmed every number in this body. It found one real hole:

F1. "No exit code is weaker than master's" was false by omission, at tools/test_ecdsa_p384_kat.py — the one wired file D3 did not touch. Round 1 did enumerate this file's exit paths; what it missed was the opt-out interaction specifically, not the file.

	--u64, U64_HOST unset
master	runs the entire VICE lane, then return 0 if total_fail == 0 else 1 → a failing emulator vector reports 1
this branch, before the fix	new gate fires before _build_prg() and before any VICE work, carrying opt_out_env="C64_ALLOW_SKIP" → with that set, 0, nothing run

The defect is scope, not arithmetic. C64_ALLOW_SKIP answers "this lane has no hardware" — precisely the operator who would set it here — and honouring it at a gate that precedes the VICE lane silences the emulator half too, which needs no hardware and could have run. They get exit 0 and never learn the emulator-only half regressed.

Fixed with opt_out_env=None rather than by moving the gate: the up-front refusal is the disclosed tightening (~0.1 s instead of hours), and moving it below the VICE lane gives that back. The remedy needs no environment variable — set U64_HOST, or drop --u64. "This lane has no hardware" is spelled by not passing --u64; asking for hardware and not supplying it is a malformed invocation, not a coverage gap to acknowledge. The unreachable backstop later in main() gets the same treatment so one file does not answer the same question two ways.

Severity was low — three deliberate operator choices, the block prints in full, and _build_prg() would likely fail first since the P-384 build has never completed — but the claim was checkable and did not hold, so the code moved rather than the claim.

F3. D5 was the one round-1 fix with no test. Nothing collects either _standalone(), so a future edit collapsing the split back to return 1 if failed else 0 would go unnoticed — in a PR whose thesis is that a helper with no call-site test is just another convention to miss. Two subprocess cases now drive the real runner in tools/test_rig_skip_contract.py with c64_test_harness made unimportable by a meta-path finder (the sibling is pip-installed, so there is no path entry to remove). That target cannot recurse — it has no spawner of its own. No opt-out → exit 2, CANNOT RUN + the vacuity string, summary still printed. C64_ALLOW_SKIP=1 → exit 0, no traceback, skipped by explicit opt-out, and the four tests that never needed the sibling still show PASS. That second case is also the D1 crash scenario, now pinned.

F2. One docstring paragraph. VoluntarySkip is an Exception, so a broad except Exception can swallow it. Not reachable today (every require() consumer names it first) and the direction is safe — a swallowed voluntary skip reports as a failure, never as a pass. The class now says it is swallowable by design and tells consumers to name it before the bare except. Deliberately not a BaseException: that is the choice that caused D1.

Round-2 red-green:

• F1 mutant — restore opt_out_env="C64_ALLOW_SKIP" on the --u64 gate → test_the_u64_gate_is_not_opt_out_able_because_it_precedes_the_vice_lane FAILS (1 failed, 11 passed).
• D5 mutant — collapse the runner to return 1 if failed or cannot else 0 → test_standalone_runner_returns_two_for_could_not_run FAILS (1 failed, 18 passed).

Review round 3 — one stale comment, fixed in 4e90ffc

Round 3 reproduced all three round-2 red-greens and confirmed every count. It found one thing, and it is a good catch: my "cannot recurse" rationale was false in the commit that wrote it.

The comment said the spawn target "has no spawner of its own". True when I reasoned it — and false by the time 76621b6 was written, because the same commit added one. test_the_u64_gate_* in test_rig_skip_contract.py spawns test_ecdsa_p384_kat.py, and _standalone() runs every test_*, so the real chain is:

pytest
  -> test_rig_skip_contract.py    (spawned by test_skip_policy.py)
       -> test_ecdsa_p384_kat.py  (spawned by its u64-gate test)

Confirmed to fire: inside the blocked standalone run the u64-gate case reports PASS, which it can only do by having spawned the KAT. No fork bomb and no hang — depth 2, terminating (the KAT spawns nothing), the whole nested run measures 0.065 s and exits 2, and the timeouts nest correctly at 180 s outer / 120 s inner.

One clause of that fix was itself wrong and is corrected in 4251c17: the comment said the chain "terminates because the P-384 KAT spawns nothing". The KAT is not a leaf in general — _build_prg() runs make clean and two make BACKEND=uci invocations at test_ecdsa_p384_kat.py:646-665. It is a leaf on this path, because the --u64 gate returns before _build_prg(). That names a coupling nobody had written down: the D5 test's cost bound rests on the F1 fix keeping that gate up front. Move it below _build_prg() — the alternative fix rejected in round 2 — and the nested run starts a full make clean && make inside the 120 s inner timeout. Two decisions one line apart, in a file that mentioned neither. Both files now say so: the chain comment names the make calls and states that moving the gate is also a decision about this test's cost, and test_rig_skip_contract.py — which adds the second link and carried no note at all — gets the chain drawn at the spawn site. A cross-file invariant documented in one of the two files is half-documented: the reader who breaks it is the one editing the other.

Nothing in the code needed to change. What needed fixing is that the comment leaned on an invariant — "no spawner exists" — which is precisely the property that would stop a third link being added later, and which was no longer true. A safety argument that has quietly gone false is worse than none, because the next reader stops looking. The comment now spells the chain out, states the depth, says the bound is by inspection with no guard in the code, gives the measured cost and the timeout nesting, and records that the earlier claim went stale inside one commit.

Merge order

#183 → #180 → #186. The ordering stands. The conflict that motivated it does not: #186 has since dropped its vendored copy of the helper entirely and rebased onto the fixed #183 tip, verified by merging three times — conflict-free, all three pytest.ini testpaths surviving, bare pytest 92 on the merged tree. Nothing is owed from this side.

Note on the build-state failures — attribution corrected

Without a build in build/, bare pytest is 10 failed, 70 passed on this branch and 10 failed, 45 passed on origin/master — the same ten, so not a regression here. The first version of this body attributed the seven test_build_flags_stamp failures to "needs ca65, that is open #177". That was wrong. ca65, ld65, ar65 and od65 were all on PATH and the seven still failed; _toolchain_missing() checks only ca65/ld65, so the #177 skip path was never taken. What fixed them was git submodule update --init libs/nistcurves libs/x25519 — a fresh worktree has no submodule working trees, so the sibling archive cannot build. #177 is a real open issue, but it is not what these failures were.

What was re-verified against current master

The commit was written at 0b55c30; #168, #172, #173, #175 and #176 have landed since. A REBASE NOTE recording this is appended to the first commit's message.

• Only one conflict on the rebase: pytest.ini testpaths, where test: pin run_all_tests.py's suite list by AST, wire the four omitted suites #175 added tools/test_runner_coverage.py. Resolved by keeping all three entries, alphabetically. Nothing else this PR touches was modified by any of the five merged PRs.
• test: an involuntary skip in test_uci_data_acc.py is a failure (#165) #172 is now MERGED, so one paragraph of the original commit message is stale, corrected in the rebase note. It called tools/test_uci_data_acc.py's missing-PRG pytest.skip "a ninth instance … owned by open PR test: an involuntary skip in test_uci_data_acc.py is a failure (#165) #172". That instance is now closed — but by a hand-rolled copy of this same policy, not by importing require(). Review round 1 sharpened this: the copy is not merely a duplicate, it diverges in four behaviours (caller-supplied C64_ALLOW_SKIP vs hardcoded C64_UCI_TESTS_OPTIONAL; .strip() == "1" vs != "1" with no strip; the standalone opt-out path; an ASSERTIONS_RUN counter this helper has no equivalent of) — and, before D1 was fixed, the copy was correct where the shared helper was not. Converging them is a separate change and a separate issue; doing it naively would have carried D1 into that module. Not attempted here.
• test_build_flags_stamp.py silently vanishes all 7 tests when ca65/ld65 are not on PATH — and it is in pytest testpaths #177 is still OPEN, so the SCOPE paragraph stands: #178b (the tools/test_pytest_boundary.py guard) still lands after it. This PR is #178a only.
• test: pin run_all_tests.py's suite list by AST, wire the four omitted suites #175's new guard does not claim the new modules: neither defines a module-level run_tests(), so tools/test_runner_coverage.py's AST check is silent about them — and it passes.
• tools/test_pytest_boundary.py passes in both directions with the two new modules listed.

Known gap, carried from the original review

The four test_macos_rig_* cases assert on source text only, so behavioural mutants of tests/rig_vice_https_macos.py survive them. Closing that needs the rig restructured to be importable without the sibling c64_test_harness, which is not a drive-by. Stated plainly: this PR pins the four bridge rigs behaviourally and the macOS rig by shape only.

No PRG bytes change: the diff is Python and pytest.ini only.

🤖 Generated with Claude Code

[JC-000]

Cross-reference: #186 (issue #177, the eighth site in #178's survey) vendors tools/_skip_policy.py byte-identical to this branch's copy — verified against origin/test/178a-skip-policy-rig-half at the time of writing — so it is green standing alone rather than importing a module not yet on master. An identical both-added file merges without conflict in either order; whichever of us lands first, the other's copy of that one file becomes a no-op.

#186 does not duplicate this PR's tools/test_skip_policy.py, tools/test_rig_skip_contract.py, rig wiring, or pytest.ini entries. It adds one pytest.ini line of its own (tools/test_flags_stamp_skip_is_loud.py); if that collides textually with this PR's two, the resolution is to keep all three lines.

If _skip_policy.py changes here before merge, say so and I will drop the file from #186 and rebase.