fix(tooling): a stale apr binary can no longer be resolved, hardcoded, or dogfooded - #2357
Merged
Conversation
…, or dogfooded Four `apr` binaries were found coexisting on the dev box: ~/.local/bin/apr 0.60.0 (no SHA) 26 days ~/.cargo/bin/apr 0.62.0 (3418754) FRESH /mnt/nvme-raid0/coverage/aprender/release/apr 0.61.0 (9514558) 1 day /mnt/nvme-raid0/targets/aprender/release/apr 0.60.0 (c55a333) 6 days This class has now caused damage three times: qwen-story-daily validated 24-day-old code while reporting green (#2344); a v0.62.0 release smoke-test read a five-hour-old binary and reported a meaningless pass; and a real product regression (#2350) stayed hidden 24 days because the nightly ran a pre-regression binary. Detection existed and was not enough. 1. RESOLUTION: ask cargo, never search. apr_bin.sh derived $APR from $CARGO_HOME/bin then PATH — both of which SEARCH, and a search finds the wrong binary eventually. It now asks `cargo metadata` for this checkout's target_directory. That matters because `.cargo/config.toml` redirects target-dir and is GITIGNORED (.gitignore:55) while its siblings .cargo/audit.toml and .cargo/mutants.toml are tracked — the directory looks version-controlled, the file that moves every build output is not. Measured: main checkout -> /mnt/nvme-raid0/coverage/aprender fresh worktree -> <worktree>/target So no absolute path is correct in both. $CARGO_HOME/bin is kept LAST (cargo install leaves nothing in target/), still freshness-checked. 2. FAIL CLOSED: `APR_BIN_STRICT=1` makes "not a git checkout" a refusal. It previously returned 0 with "freshness not asserted" — a fail-OPEN hole in a script whose only job is refusing unproven binaries. 3. HARDCODED ABSOLUTE PATHS are now a build failure. check_apr_bin_pinned.sh treated `/mnt/.../target/release/apr` as PINNED because it ends in target/release/apr — so the exact path that was 6 days and two minor versions stale passed the guard. New ABS-APR class, mutation-verified by injecting that literal path into qwen-story.sh: RED exit 1, GREEN exit 0 after revert. The regex is anchored at the path START; the first draft matched the `/apr` inside RELATIVE `target/release/apr` and flagged correct code. Verified against a 12-case table (4 must match, 8 must not) — this regex class has now been gotten wrong four times here, so the table ships with it. 4. THE DOGFOOD PROTOCOL certified releases with an unpinned binary: - Gate 1 ran `cargo install --path` (writes $CARGO_HOME/bin) and then a BARE `apr`, which PATH resolved to the 26-day-old ~/.local/bin copy. - Gate 13, the protocol's ONLY freshness assertion, SKIPPED when the binary had no embedded SHA "(likely crates.io install)". `~/.local/bin/apr` reports exactly `apr 0.60.0 (v0.60.0+no-git)`. The one artifact most likely to be executed was the one that made the only gate excuse itself. Both now go through `. scripts/apr_bin.sh || exit 1`; no-SHA is a FAIL unless DOGFOOD_ALLOW_UNPINNED=1 is set deliberately. 5. CLAUDE.md's "MANDATORY: use apr tools first" block — the most-followed instruction in the repo — told every agent to run bare `apr qa`. It now pins Step 0 and uses "$APR" throughout. Guards green: check_apr_bin_pinned, check_sourced_libs_option_neutral, check_pass_grep_anchored. bashrs errors across both scripts: 8 -> 5. NOT fixed here: the stale binaries themselves still exist on the box (deleting another user's ~/.local/bin is an operator decision, not a repo change), and historical specs under docs/specifications/ still quote the old absolute path as a record of past runs. Both tracked separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four
aprbinaries were coexisting on the dev box:~/.local/bin/apr~/.cargo/bin/apr341875412)/mnt/nvme-raid0/coverage/…/apr95145584f)/mnt/nvme-raid0/targets/…/aprc55a3335c)This class has caused damage three times:
qwen-story-dailyvalidated 24-day-old code while reporting green (#2344); the v0.62.0 release smoke-test read a five-hour-old binary and reported a meaningless pass; and a real product regression (#2350) stayed hidden for 24 days because the nightly ran a pre-regression binary.Detection already existed. It was not enough.
1. Resolution: ask cargo, never search
apr_bin.shderived$APRfrom$CARGO_HOME/binthen PATH — both of which search, and a search finds the wrong binary eventually. It now askscargo metadatafor this checkout'starget_directory.That matters because
.cargo/config.tomlredirects target-dir and is gitignored (.gitignore:55) while its siblings.cargo/audit.tomland.cargo/mutants.tomlare tracked — the directory looks version-controlled; the file that moves every build output is not. Measured:No absolute path is correct in both.
$CARGO_HOME/binis kept last (cargo install leaves nothing intarget/), still freshness-checked.2. Fail closed
APR_BIN_STRICT=1makes "not a git checkout" a refusal. It previously returned 0 with "freshness not asserted" — a fail-open hole in a script whose only job is refusing unproven binaries.3. Hardcoded absolute paths are now a build failure
check_apr_bin_pinned.shtreated/mnt/.../target/release/apras pinned, because it ends intarget/release/apr. So the exact path that was 6 days and two minor versions stale sailed through the guard.New
ABS-APRclass, mutation-verified by injecting that literal path intoqwen-story.sh: RED exit 1, GREEN exit 0 after revert.The regex is anchored at the path start — the first draft matched the
/aprinside relativetarget/release/aprand flagged correct code. Verified against a 12-case table (4 must match, 8 must not). This regex class has now been gotten wrong four times in this repo, so the table ships with it.4. The dogfood protocol was certifying releases with an unpinned binary
This is the part that matters most, and it was found by audit rather than by me:
cargo install --path(writes$CARGO_HOME/bin) and then a bareapr— PATH resolved to the 26-day-old~/.local/bincopy.~/.local/bin/aprreports exactlyapr 0.60.0 (v0.60.0+no-git).So the single artifact most likely to be executed was also the one that made the only freshness gate excuse itself. A dogfood run is worthless if it exercised a binary other than the one being shipped.
Both gates now go through
. scripts/apr_bin.sh || exit 1; no-SHA is a FAIL unlessDOGFOOD_ALLOW_UNPINNED=1is set deliberately.5. CLAUDE.md
The "MANDATORY: use apr tools first" block — the most-followed instruction in the repo — told every agent to run bare
apr qa. It now pins Step 0 and uses"$APR"throughout.Verification
check_apr_bin_pinned,check_sourced_libs_option_neutral,check_pass_grep_anchoredall exit 0. bashrs errors across both changed scripts: 8 → 5.Not fixed here
The stale binaries still exist on the box — deleting a developer's
~/.local/bin/apris an operator decision, not a repo change. Historical specs underdocs/specifications/still quote the old absolute path as a record of past runs; those are records, not instructions. Both tracked separately.🤖 Generated with Claude Code