fix(ci): qwen-story-daily installed a fresh apr and then ran a 24-day-old one - #2344
Conversation
…-old one
Every beat in the daily story has been validating stale code. On the cuda
runner:
/home/noah/.local/bin/apr 0.60.0 2026-07-06 <- PATH pos 2, WINS
/home/noah/.cargo/bin/apr 0.61.0 (e514cc5) 2026-07-30 <- PATH pos 8
`cargo install --path crates/apr-cli --force` writes to ~/.cargo/bin, the step
then reports success ("Replacing /home/noah/.cargo/bin/apr"), and
scripts/qwen-story.sh invokes bare `apr` - which resolves to the 24-day-old
0.60.0 binary in ~/.local/bin. Fresh install, stale execution, green result.
HOW IT SURFACED, because this is the useful part. The story's own log said:
PASS B2 format_parity (GGUF vs SafeTensors, executed: false true 17 17)
`17 17` is value/threshold from the OLD gate - a token id compared against
itself. The gate merged in #2337 reports matched/total DECODE STEPS, i.e.
`64 64`, and `git cat-file -p e514cc5:...forward_error.rs` confirms
lockstep_decode_parity IS in the commit the run checked out. Source new, binary
old. Confirmed directly: `apr --version` printed 0.60.0 while Cargo.toml says
0.61.0.
So the leg added in #2337 did execute and did report PASS - against July 6 code
that cannot perform a 64-step decode. A green that proved nothing, which is
precisely the class this story is supposed to catch in others.
TWO CHANGES, because the path fix alone would rot again:
1. Prepend $HOME/.cargo/bin to GITHUB_PATH after install, so the story runs the
binary it just built.
2. Fail closed on staleness: compare `apr --version` against the workspace
version parsed from Cargo.toml and `exit 1` with `::error::` if they differ.
A PATH fix is a point fix; this asserts the property. Verified against both
binaries actually present on the runner:
workspace WANT=0.61.0
STALE -> would FAIL : apr 0.60.0 (v0.60.0+no-git) (~/.local/bin/apr)
FRESH -> would PASS : apr 0.61.0 (e514cc5) (~/.cargo/bin/apr)
Scanned the other workflows for the same shape: book.yml also does
`cargo install --path crates/apr-cli`, but scripts/check_book_cli_parity.sh
already defaults APR to an explicit ~/.cargo/bin/apr, so it is unaffected.
cuda-nightly.yml passes an explicit APR_BIN. This was the only instance.
Contract: FALSIFY-QWEN-STORY-011, executed. pv validate: 0 errors, 0 warnings.
NOTE the stale ~/.local/bin/apr is still on the runner; this change makes the
story ignore it rather than removing it, since deleting a binary from the
operator's host is their call, not CI's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the first commit on this branch, which fixed qwen-story-daily's PATH and asserted the version. That was a point fix; this makes the property hold everywhere and makes reintroduction fail the PR that does it. Three layers. 1. scripts/apr_bin.sh - resolve the binary and PROVE its provenance. crates/apr-cli/build.rs already embeds the build-time git SHA (contract apr-version-traceability-v1, F-VERSION-001..004), so `apr --version` prints `apr 0.61.0 (e514cc5)`. "Was this built from HEAD?" is therefore a string comparison, not a guess. Sourcing the script exports $APR and hard-fails on a mismatch, listing EVERY apr on PATH in resolution order so the shadowing is visible rather than something the reader must go find: STALE apr BINARY resolved : /home/noah/.cargo/bin/apr reports : apr 0.61.0 (e514cc5) HEAD : 94649c0 every apr on PATH (first wins): /home/noah/.local/bin/apr apr 0.60.0 (v0.60.0+no-git) /home/noah/.cargo/bin/apr apr 0.61.0 (e514cc5) Version alone would not be enough: a rebuilt-but-not-reinstalled binary carries the same version and a different SHA. 2. scripts/qwen-story.sh sources it. run_cmd substitutes a leading bare `apr` with "$APR", which pins all 16 call sites without touching them, and the backgrounded `apr serve run` at :302 - which bypassed run_cmd entirely and was found by the checker below, not by reading - is pinned explicitly. 3. scripts/check_apr_bin_pinned.sh, wired per-PR into ci.yml's guard-runner-labels (which `gate` needs). Detection tells you after the fact; this makes a new bare `apr` on a CI surface fail the PR. Scope is CI surfaces only - workflows plus scripts a workflow actually names, discovered by grep rather than hand-listed so newly wired scripts are covered automatically. Developer scripts CI never runs are deliberately out of scope; the invariant is "what CI executes was built from the commit under test", not "nobody may type apr". THE REGEX TOOK THREE TRIES, and both wrong versions were caught by mutation rather than by review, which is the whole argument for mutation-testing a gate: v1 anchored on shell separators (^ ; & | && || $()) and MISSED `- run: apr qa model.gguf` - a colon is not a separator. It would have passed the exact workflow line it exists to catch. v2 scanned line content and flagged TEN step names reading "Pillar-1 - apr vs scikit-learn ...". A gate that fires on its own labels gets disabled, which is worse than no gate. v3 anchors on COMMAND POSITION - line start, after a shell separator, or after a YAML `run:`. Verified against an 8-case table covering both directions (bare-at-line-start, after &&, YAML run:, vs. step names, quoted prose, "$APR", explicit paths). Verified in five directions: baseline exit 0, "26 CI-surface file(s) scanned" mutation A bare `apr` in a workflow -> exit 1 mutation B bare `apr` in a CI script -> exit 1 restored exit 0 fail-closed MIN_EXPECTED unmet -> exit 1, not OK-on-empty-set Contract: FALSIFY-QWEN-STORY-012/013, both executed. pv validate 0/0. bashrs: 0 errors on all three scripts. Story falsifiers 002/003/004 still hold (8 beats, 8 pmat_hunt calls, run_cmd shape preserved). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second commit: from detectable to structurally impossibleThe first commit fixed this one workflow's PATH and asserted its version. That's a point fix — it rots the next time PATH changes, and it does nothing for any other surface. Three layers added: 1. Version alone would be insufficient — a rebuilt-but-not-reinstalled binary carries the same version and a different SHA. 2. 3. The regex took three tries, and both wrong versions were caught by mutation
Neither wrong version was caught by review. That's the argument for mutation-testing a gate before trusting it. Verified five ways: baseline exit 0 (26 files scanned) · bare
|
Every beat in the daily story has been validating stale code.
cargo install --path crates/apr-cli --forcewrites to~/.cargo/bin, the step reports success ("Replacing /home/noah/.cargo/bin/apr"), andscripts/qwen-story.shthen invokes bareapr— which resolves to the 24-day-old 0.60.0 binary in~/.local/bin. Fresh install, stale execution, green result.How it surfaced
The story's own log said:
17 17is value/threshold from the old gate — a token id compared against itself. The gate merged in #2337 reports matched/total decode steps, i.e.64 64. Andgit cat-file -p e514cc5ed:...forward_error.rsconfirmslockstep_decode_parityis in the commit that run checked out.Source new, binary old. Confirmed directly:
apr --versionprinted0.60.0whileCargo.tomlsays0.61.0.So the leg added in #2337 did execute and did report PASS — against July 6 code that cannot perform a 64-step decode. A green that proved nothing, which is exactly the class this story exists to catch in others.
Two changes, because the path fix alone would rot again
Prepend
$HOME/.cargo/bintoGITHUB_PATHafter install, so the story runs the binary it just built.Fail closed on staleness — compare
apr --versionagainst the workspace version parsed fromCargo.toml,exit 1with::error::if they differ. A PATH fix is a point fix; this asserts the property. Verified against both binaries actually present on the runner:Scope
Scanned the other workflows for the same shape.
book.ymlalso runscargo install --path crates/apr-cli, butscripts/check_book_cli_parity.shalready defaultsAPRto an explicit~/.cargo/bin/apr, so it's unaffected.cuda-nightly.ymlpasses an explicitAPR_BIN. This was the only instance.Contract:
FALSIFY-QWEN-STORY-011, executed.pv validate: 0 errors, 0 warnings.The stale
~/.local/bin/apris still on the runner; this makes the story ignore it rather than removing it — deleting a binary from the operator's host is their call, not CI's.🤖 Generated with Claude Code