Skip to content

fix(ci): qwen-story-daily installed a fresh apr and then ran a 24-day-old one - #2344

Merged
noahgift merged 2 commits into
mainfrom
fix/qwen-story-stale-apr-binary
Jul 30, 2026
Merged

fix(ci): qwen-story-daily installed a fresh apr and then ran a 24-day-old one#2344
noahgift merged 2 commits into
mainfrom
fix/qwen-story-stale-apr-binary

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Every beat in the daily story has been validating stale code.

/home/noah/.local/bin/apr   0.60.0             2026-07-06   <- PATH pos 2, WINS
/home/noah/.cargo/bin/apr   0.61.0 (e514cc5ed) 2026-07-30   <- PATH pos 8

cargo install --path crates/apr-cli --force writes to ~/.cargo/bin, the step reports success ("Replacing /home/noah/.cargo/bin/apr"), and scripts/qwen-story.sh then invokes bare apr — which resolves to the 24-day-old 0.60.0 binary in ~/.local/bin. Fresh install, stale execution, green result.

How it surfaced

The story's own log said:

PASS  B2 format_parity (GGUF vs SafeTensors, executed: false true 17 17)

17 17 is value/threshold from the old gate — a token id compared against itself. The gate merged in #2337 reports matched/total decode steps, i.e. 64 64. And git cat-file -p e514cc5ed:...forward_error.rs confirms lockstep_decode_parity is in the commit that run checked out.

Source new, binary old. Confirmed directly: apr --version printed 0.60.0 while Cargo.toml says 0.61.0.

So the leg added in #2337 did execute and did report PASS — against July 6 code that cannot perform a 64-step decode. A green that proved nothing, which is exactly the class this story exists to catch in others.

Two changes, because the path fix alone would rot again

  1. Prepend $HOME/.cargo/bin to GITHUB_PATH after install, so the story runs the binary it just built.

  2. Fail closed on staleness — compare apr --version against the workspace version parsed from Cargo.toml, exit 1 with ::error:: if they differ. A PATH fix is a point fix; this asserts the property. Verified against both binaries actually present on the runner:

workspace WANT=0.61.0
STALE -> would FAIL : apr 0.60.0 (v0.60.0+no-git)  (~/.local/bin/apr)
FRESH -> would PASS : apr 0.61.0 (e514cc5ed)       (~/.cargo/bin/apr)

Scope

Scanned the other workflows for the same shape. book.yml also runs cargo install --path crates/apr-cli, but scripts/check_book_cli_parity.sh already defaults APR to an explicit ~/.cargo/bin/apr, so it's unaffected. cuda-nightly.yml passes an explicit APR_BIN. This was the only instance.

Contract: FALSIFY-QWEN-STORY-011, executed. pv validate: 0 errors, 0 warnings.

The stale ~/.local/bin/apr is still on the runner; this makes the story ignore it rather than removing it — deleting a binary from the operator's host is their call, not CI's.

🤖 Generated with Claude Code

…-old one

Every beat in the daily story has been validating stale code. On the cuda
runner:

    /home/noah/.local/bin/apr   0.60.0             2026-07-06   <- PATH pos 2, WINS
    /home/noah/.cargo/bin/apr   0.61.0 (e514cc5) 2026-07-30   <- PATH pos 8

`cargo install --path crates/apr-cli --force` writes to ~/.cargo/bin, the step
then reports success ("Replacing /home/noah/.cargo/bin/apr"), and
scripts/qwen-story.sh invokes bare `apr` - which resolves to the 24-day-old
0.60.0 binary in ~/.local/bin. Fresh install, stale execution, green result.

HOW IT SURFACED, because this is the useful part. The story's own log said:

    PASS  B2 format_parity (GGUF vs SafeTensors, executed: false true 17 17)

`17 17` is value/threshold from the OLD gate - a token id compared against
itself. The gate merged in #2337 reports matched/total DECODE STEPS, i.e.
`64 64`, and `git cat-file -p e514cc5:...forward_error.rs` confirms
lockstep_decode_parity IS in the commit the run checked out. Source new, binary
old. Confirmed directly: `apr --version` printed 0.60.0 while Cargo.toml says
0.61.0.

So the leg added in #2337 did execute and did report PASS - against July 6 code
that cannot perform a 64-step decode. A green that proved nothing, which is
precisely the class this story is supposed to catch in others.

TWO CHANGES, because the path fix alone would rot again:

1. Prepend $HOME/.cargo/bin to GITHUB_PATH after install, so the story runs the
   binary it just built.

2. Fail closed on staleness: compare `apr --version` against the workspace
   version parsed from Cargo.toml and `exit 1` with `::error::` if they differ.
   A PATH fix is a point fix; this asserts the property. Verified against both
   binaries actually present on the runner:

     workspace WANT=0.61.0
     STALE -> would FAIL : apr 0.60.0 (v0.60.0+no-git)  (~/.local/bin/apr)
     FRESH -> would PASS : apr 0.61.0 (e514cc5)       (~/.cargo/bin/apr)

Scanned the other workflows for the same shape: book.yml also does
`cargo install --path crates/apr-cli`, but scripts/check_book_cli_parity.sh
already defaults APR to an explicit ~/.cargo/bin/apr, so it is unaffected.
cuda-nightly.yml passes an explicit APR_BIN. This was the only instance.

Contract: FALSIFY-QWEN-STORY-011, executed. pv validate: 0 errors, 0 warnings.

NOTE the stale ~/.local/bin/apr is still on the runner; this change makes the
story ignore it rather than removing it, since deleting a binary from the
operator's host is their call, not CI's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge July 30, 2026 08:36
Follow-up to the first commit on this branch, which fixed qwen-story-daily's
PATH and asserted the version. That was a point fix; this makes the property
hold everywhere and makes reintroduction fail the PR that does it.

Three layers.

1. scripts/apr_bin.sh - resolve the binary and PROVE its provenance.
   crates/apr-cli/build.rs already embeds the build-time git SHA (contract
   apr-version-traceability-v1, F-VERSION-001..004), so `apr --version` prints
   `apr 0.61.0 (e514cc5)`. "Was this built from HEAD?" is therefore a string
   comparison, not a guess. Sourcing the script exports $APR and hard-fails on
   a mismatch, listing EVERY apr on PATH in resolution order so the shadowing
   is visible rather than something the reader must go find:

       STALE apr BINARY
         resolved : /home/noah/.cargo/bin/apr
         reports  : apr 0.61.0 (e514cc5)
         HEAD     : 94649c0
         every apr on PATH (first wins):
           /home/noah/.local/bin/apr    apr 0.60.0 (v0.60.0+no-git)
           /home/noah/.cargo/bin/apr    apr 0.61.0 (e514cc5)

   Version alone would not be enough: a rebuilt-but-not-reinstalled binary
   carries the same version and a different SHA.

2. scripts/qwen-story.sh sources it. run_cmd substitutes a leading bare `apr`
   with "$APR", which pins all 16 call sites without touching them, and the
   backgrounded `apr serve run` at :302 - which bypassed run_cmd entirely and
   was found by the checker below, not by reading - is pinned explicitly.

3. scripts/check_apr_bin_pinned.sh, wired per-PR into ci.yml's
   guard-runner-labels (which `gate` needs). Detection tells you after the
   fact; this makes a new bare `apr` on a CI surface fail the PR. Scope is
   CI surfaces only - workflows plus scripts a workflow actually names,
   discovered by grep rather than hand-listed so newly wired scripts are
   covered automatically. Developer scripts CI never runs are deliberately out
   of scope; the invariant is "what CI executes was built from the commit under
   test", not "nobody may type apr".

THE REGEX TOOK THREE TRIES, and both wrong versions were caught by mutation
rather than by review, which is the whole argument for mutation-testing a gate:

  v1 anchored on shell separators (^ ; & | && || $()) and MISSED
     `- run: apr qa model.gguf` - a colon is not a separator. It would have
     passed the exact workflow line it exists to catch.
  v2 scanned line content and flagged TEN step names reading
     "Pillar-1 - apr vs scikit-learn ...". A gate that fires on its own labels
     gets disabled, which is worse than no gate.
  v3 anchors on COMMAND POSITION - line start, after a shell separator, or
     after a YAML `run:`. Verified against an 8-case table covering both
     directions (bare-at-line-start, after &&, YAML run:, vs. step names,
     quoted prose, "$APR", explicit paths).

Verified in five directions:
  baseline      exit 0, "26 CI-surface file(s) scanned"
  mutation A    bare `apr` in a workflow  -> exit 1
  mutation B    bare `apr` in a CI script -> exit 1
  restored      exit 0
  fail-closed   MIN_EXPECTED unmet -> exit 1, not OK-on-empty-set

Contract: FALSIFY-QWEN-STORY-012/013, both executed. pv validate 0/0.
bashrs: 0 errors on all three scripts. Story falsifiers 002/003/004 still hold
(8 beats, 8 pmat_hunt calls, run_cmd shape preserved).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Second commit: from detectable to structurally impossible

The first commit fixed this one workflow's PATH and asserted its version. That's a point fix — it rots the next time PATH changes, and it does nothing for any other surface. Three layers added:

1. scripts/apr_bin.sh — prove provenance, don't guess it.
crates/apr-cli/build.rs already embeds the build-time git SHA (apr-version-traceability-v1), so apr --version prints apr 0.61.0 (e514cc5ed). "Was this built from HEAD?" is a string comparison. Sourcing exports $APR and hard-fails on mismatch, listing every apr on PATH in resolution order:

STALE apr BINARY
  resolved : /home/noah/.cargo/bin/apr
  reports  : apr 0.61.0 (e514cc5ed)
  HEAD     : 94649c00f
  every apr on PATH (first wins):
    /home/noah/.local/bin/apr    apr 0.60.0 (v0.60.0+no-git)
    /home/noah/.cargo/bin/apr    apr 0.61.0 (e514cc5ed)

Version alone would be insufficient — a rebuilt-but-not-reinstalled binary carries the same version and a different SHA.

2. qwen-story.sh sources it. run_cmd substitutes a leading bare apr with "$APR", pinning all 16 call sites at once. The backgrounded apr serve run at :302 bypassed run_cmd entirely — found by the checker, not by reading — and is pinned explicitly.

3. scripts/check_apr_bin_pinned.sh, wired per-PR into guard-runner-labels. Detection tells you afterwards; this fails the PR that reintroduces a bare apr. Scope is CI surfaces only — workflows plus scripts a workflow actually names, discovered by grep so newly wired scripts are covered automatically.

The regex took three tries, and both wrong versions were caught by mutation

outcome
v1 — anchored on shell separators missed - run: apr qa model.gguf (a colon isn't a separator) — would have passed the exact line it exists to catch
v2 — scanned line content flagged ten step names reading "Pillar-1 — apr vs scikit-learn …". A gate that fires on its own labels gets disabled, which is worse than no gate
v3 — anchors on command position verified against an 8-case table covering both directions

Neither wrong version was caught by review. That's the argument for mutation-testing a gate before trusting it.

Verified five ways: baseline exit 0 (26 files scanned) · bare apr in a workflow → exit 1 · bare apr in a CI script → exit 1 · restored → exit 0 · MIN_EXPECTED unmet → exit 1, not OK-on-empty-set.

FALSIFY-QWEN-STORY-012/013 executed; pv validate 0/0; bashrs 0 errors on all three scripts; story falsifiers 002/003/004 still hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant