TEST-GUARD-GATE-EVIDENCE declares a step that cannot run from a checkout, with a stale threshold

[avrabe]

Found while implementing #403, by executing every declared verification step
rather than reading them. 17 of 18 pass; this is the one that does not.

artifacts/verification.yaml, TEST-GUARD-GATE-EVIDENCE (status: implemented):

    fields:
      method: automated-test
      steps:
        - run: tools/check_mutants_report.py --self-test
        - run: tools/check_mutants_report.py --output-dir mutants-out --max-missed 210

Executed from a clean checkout:

ok   [0] tools/check_mutants_report.py --self-test
FAIL [1] tools/check_mutants_report.py --output-dir mutants-out --max-missed 210
       ::error::no cargo-mutants report directory at mutants-out/mutants.out —
       the run did not complete. ...
       mutants-out/ does not exist at all.

Two distinct problems

1. The step cannot run outside its CI job. mutants-out/ only exists after a
~3-hour cargo mutants run in the mutants job. tools/run_verification.py
executes declared steps and scores returncode == 0, so this artifact is
permanently FAILED for anyone who runs the verification gate over it. That is the
mirror image of #403: there, a claiming artifact had no step and was silently
skipped; here it has a step that cannot succeed.

2. The threshold is stale. It says --max-missed 210; .github/workflows/ci.yml
uses 292, and the live run on main reports 291 survivors. So the artifact
records a bound that has not been the enforced one since #389/#401 landed. Nothing
cross-checks the number in the artifact against the number in the workflow — the
same duplicated-fact shape as #381, where the report path was written twice and the
copies disagreed for four months.

Why it is not simply deleted

Removing the step would make the artifact pass, which is exactly the move
REQ-GUARD-GATE-EVIDENCE-002/003 exist to prevent. The honest options are:

1. Point it at something runnable — the self-test already is; a second step could
   assert the workflow declares a threshold, rather than re-running the gate.
2. Or keep the live invocation and mark the artifact so the verification runner
   knows it is CI-context-only — but no such concept exists today, and inventing
   one is an exemption mechanism, which Lean 'Fail on sorry' gate enforces zero obligations: all 12 sorrys are self-exempted by '-- TODO' #385 is a cautionary tale about.
3. And regardless: single-source the threshold so the artifact and ci.yml cannot
   disagree.

Note the general shape for whoever picks this up: --max-missed 210 in an
artifact and --max-missed 292 in a workflow is a number stated twice. Every
other instance of that in this repo has eventually drifted.

Found by running the steps, not reading them. A declared step nobody has executed
is a claim with no gate on it.

🤖 Generated with Claude Code

[avrabe]

spar issue-hunt run — picked up #414, PR #423 opened (CI pending).

What shipped

• REQ-GUARD-GATE-EVIDENCE-004 (new, status: implemented, release: v0.38.0, not human-scoped) — the two obligations this issue names: a threshold a gate enforces shall live in one place, and an evidence step shall run from a clean checkout.
• tools/check_mutants_report.py --assert-workflow-threshold WORKFLOW — reads the committed workflow (no mutants run), asserts exactly one bounded, un-neutered mutation gate, prints the threshold. stdlib-only.
• TEST-GUARD-GATE-EVIDENCE step 2 retargeted from --output-dir mutants-out --max-missed 210 → --assert-workflow-threshold .github/workflows/ci.yml. The number now lives once (in ci.yml); the artifact reads it rather than restating it.

Both problems you raised are closed: the artifact is now runnable from a checkout (run_verification.py scores it [ OK ], was permanently FAILED), and the 210-vs-292 duplicated fact is structurally gone.

Non-vacuity (executed, not prose)

Decision cases folded into --self-test (34/0): a different threshold prints differently, and absent / neutered (|| true and || :) / non-integer / doubly-declared-disagreeing gates each fail. Every guard branch is load-bearing — mutating any one flips a self-test case to FAIL.

Clean-room outcome

An independent adversarial reviewer re-derived all claims from a clean checkout. It confirmed A–F and caught one genuine defect: the || : shell no-op was silently accepted (a stray \b after the : alternative in the neuter regex). Fixed in this PR, with a self-test case that now covers it. No over-claims survived: REQ-004 is implemented not verified, and it explicitly does not claim the threshold value is correct or that cargo-mutants measured the workspace.

NOT claimed

That 292 is the right survivor count (#382's ratchet), nor that the mutation run measures the workspace correctly (REQ-001's honest ceiling). Only that the number lives once and the step runs.

Handoff / release note

PR #423 required checks are pending (Mutation Testing ~2h, Lean ~90m). A later run advances it via the overlap guard. Separately: v0.37.0 is currently staged but untagged — #412's bump is merged to main (da9b281) yet the latest tag/release is v0.36.0, so the signed v0.37.0 tag is still pending on the maintainer. I targeted REQ-004 at v0.38.0 and did not cut a version bump this run, to avoid jumping ahead of that in-flight release; the bump/tag for this work should follow once v0.37.0 is tagged (or the maintainer may re-cut, as was done for #411/v0.36.0).

---

Generated by Claude Code

[avrabe]

Merged. PR #423 landed on main as 1998932 — all 21 check runs green, including Mutation Testing (passed in ~53m) and Lean proof typecheck; mergeable_state: clean, squash-merged, branch deleted. The clean-room-found || : neuter gap was fixed before merge.

TEST-GUARD-GATE-EVIDENCE now scores [ OK ] from a clean checkout, and the mutation threshold lives once (in ci.yml). This issue is closed by the merge.

No release cut this run, by design: v0.37.0 is still staged-but-untagged (#412's bump is on main at da9b281, latest tag is v0.36.0), so cutting a v0.38.0 bump ahead of it would jump the maintainer's in-flight release. REQ-GUARD-GATE-EVIDENCE-004 stays status: implemented targeting v0.38.0; its promotion to verified + the version bump/signed tag should follow once v0.37.0 is tagged (git tag -s v0.37.0 && git push origin v0.37.0), or be folded in if v0.37.0 is re-cut.

---

Generated by Claude Code