flags-stamp suite: an absent cc65 must be loud, not green (#177)

[JC-000]

Closes #177. The eighth site in #178's survey.

Merge order is load-bearing

#183  (this PR's base — issue #174)
  ->  #180  (#178a, brings tools/_skip_policy.py)
  ->  #186  (this PR)

This branch imports tools/_skip_policy.py and no longer carries a copy.
An earlier revision vendored it, on the reasoning that an identical both-added
file merges cleanly in either order. #180 then amended the module and
git merge-tree reported an add/add conflict in both directions, so that claim
is retracted and the copy is dropped instead. No sha is quoted here on
purpose: the module has moved three times while this PR was open, which is
itself the argument for importing rather than vendoring. The authority is
origin/test/178a-skip-policy-rig-half, whatever it currently points at.

Dropping the copy costs nothing — this branch uses only require(),
cannot_run() and EXIT_CANNOT_RUN, whose contracts have not changed through
any of those revisions.

Verified against that branch's current head: the merge is conflict-free
(including pytest.ini, which ends up with all three new testpaths lines),
the merged tree's module is byte-identical to the one #180 provides — so it
comes from exactly one place — and on that tree
python3 tools/test_flags_stamp_skip_is_loud.py is PASSED: 0 failure(s) (4 executed) with bare pytest at 92 passed. Until #180 lands, this
branch alone is 1 error and cannot import; every measurement below was taken
on the merged tree.

The defect

Eight tests, each shelling out to make, each opening with

    missing = _toolchain_missing()
    if missing:
        print(f"SKIP: {missing} not on PATH")
        return

A pytest test that returns None without asserting is a pass, and the
module is pinned in pytest.ini testpaths — so on a machine without cc65
bare pytest at the repo root goes green while asserting the
build/flags.stamp invariant it never checked. The SKIP: lines are not even
visible: pytest captures stdout on a passing test and discards it.

Red / green

New tools/test_flags_stamp_skip_is_loud.py re-runs the subject module in a
subprocess whose PATH has had every directory containing ca65/ld65
removed — computed from the real PATH, not a hardcoded /usr/bin:/bin,
so it strips the toolchain wherever it happens to be installed.

RED, subject module at the pre-fix revision:

=== involuntary skip must be loud (#177) ===
  FAIL test_opt_out_is_explicit_and_still_warns
  FAIL test_pytest_run_without_toolchain_is_not_green
  FAIL test_standalone_run_without_toolchain_exits_cannot_run
  ok   test_the_strip_actually_strips

FAILED: 3 failure(s) (4 executed)

with the subject's own output captured in the failure message — the exact shape
#177 reports (SKIP: ca65 not on PATH / ok × 8, then PASSED: 0 failure(s)).

GREEN, after the fix: PASSED: 0 failure(s) (4 executed), and identically
against #180's amended module.

Both channels after the fix, toolchain stripped:

$ python3 tools/test_build_flags_stamp.py ; echo $?
============================================================
COULD NOT RUN: ca65 not on PATH -- every case here shells out to `make`
  0 of 9 checks executed
  this run certifies NOTHING about build/flags.stamp — …
  set C64_ALLOW_SKIP=1 to accept an unverified run (exit 0 instead of 2)
  exit 2 = could not run (1 would mean a check failed)
============================================================
2

$ pytest -q tools/test_build_flags_stamp.py
9 failed in 0.11s

$ C64_ALLOW_SKIP=1 pytest -q tools/test_build_flags_stamp.py
SKIPPED [9] tools/_skip_policy.py: COULD NOT RUN: ca65 not on PATH … --
  this run certifies NOTHING about build/flags.stamp … [C64_ALLOW_SKIP=1 set]
9 skipped in 0.02s

With cc65 present nothing changes: bare pytest at the repo root is
92 passed on the merged tree described above.

Shape of the fix

lane	before	after
pytest	bare return → pass	require() raises SkipPolicyError; the reason string carries the vacuity warning, because under -ra the reason is the only surviving channel
standalone	PASSED: 0 failure(s), exit 0	cannot_run() → exit 2, distinct from 1 ("a check ran and failed"), plus a ran + failed == 0 guard so PASSED can never be printed by a run that executed nothing
opt-out	none	C64_ALLOW_SKIP=1 buys exit 0 / a real pytest skip and still prints the warning; exactly "1", so C64_ALLOW_SKIP=0 does not open the hatch (asserted)

main() hoists the toolchain check out of the loop and calls cannot_run()
directly, rather than letting eight require() calls report eight identical
failures for one missing binary. Review confirmed this is also why the
standalone lane's behaviour is unaffected by the _skip_policy.py revision:
it never reaches require().

test_the_strip_actually_strips guards the guard — a PATH-stripping bug
would make every other assertion run against a machine that can build, and
they would all pass having tested nothing: the same vacuity, one level up.

Known follow-up, not a blocker

_toolchain_missing() checks only ca65/ld65. An uninitialised submodule
(no libs/nistcurves working tree) is a different failure: farm.make()
asserts and the suite reports eight loud failures rather than a named
COULD NOT RUN. Loud is the correct half; the naming is not. Worth a separate
pass, and it does not affect what this PR pins — #177's own repro is verbatim a
PATH strip producing seven silent passes.

🤖 Generated with Claude Code