Skip to content

fix(tooling): a stale apr binary can no longer be resolved, hardcoded, or dogfooded - #2357

Merged
noahgift merged 1 commit into
mainfrom
fix/apr-binary-ambiguity
Aug 1, 2026
Merged

fix(tooling): a stale apr binary can no longer be resolved, hardcoded, or dogfooded#2357
noahgift merged 1 commit into
mainfrom
fix/apr-binary-ambiguity

Conversation

@noahgift

@noahgift noahgift commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Four apr binaries were coexisting on the dev box:

path version age
~/.local/bin/apr 0.60.0 (no SHA) 26 days
~/.cargo/bin/apr 0.62.0 (341875412) fresh
/mnt/nvme-raid0/coverage/…/apr 0.61.0 (95145584f) 1 day
/mnt/nvme-raid0/targets/…/apr 0.60.0 (c55a3335c) 6 days

This class has caused damage three times: qwen-story-daily validated 24-day-old code while reporting green (#2344); the v0.62.0 release smoke-test read a five-hour-old binary and reported a meaningless pass; and a real product regression (#2350) stayed hidden for 24 days because the nightly ran a pre-regression binary.

Detection already existed. It was not enough.

1. Resolution: ask cargo, never search

apr_bin.sh derived $APR from $CARGO_HOME/bin then PATH — both of which search, and a search finds the wrong binary eventually. It now asks cargo metadata for this checkout's target_directory.

That matters because .cargo/config.toml redirects target-dir and is gitignored (.gitignore:55) while its siblings .cargo/audit.toml and .cargo/mutants.toml are tracked — the directory looks version-controlled; the file that moves every build output is not. Measured:

main checkout   -> /mnt/nvme-raid0/coverage/aprender
fresh worktree  -> <worktree>/target

No absolute path is correct in both. $CARGO_HOME/bin is kept last (cargo install leaves nothing in target/), still freshness-checked.

2. Fail closed

APR_BIN_STRICT=1 makes "not a git checkout" a refusal. It previously returned 0 with "freshness not asserted" — a fail-open hole in a script whose only job is refusing unproven binaries.

3. Hardcoded absolute paths are now a build failure

check_apr_bin_pinned.sh treated /mnt/.../target/release/apr as pinned, because it ends in target/release/apr. So the exact path that was 6 days and two minor versions stale sailed through the guard.

New ABS-APR class, mutation-verified by injecting that literal path into qwen-story.sh: RED exit 1, GREEN exit 0 after revert.

The regex is anchored at the path start — the first draft matched the /apr inside relative target/release/apr and flagged correct code. Verified against a 12-case table (4 must match, 8 must not). This regex class has now been gotten wrong four times in this repo, so the table ships with it.

4. The dogfood protocol was certifying releases with an unpinned binary

This is the part that matters most, and it was found by audit rather than by me:

  • Gate 1 ran cargo install --path (writes $CARGO_HOME/bin) and then a bare apr — PATH resolved to the 26-day-old ~/.local/bin copy.
  • Gate 13, the protocol's only freshness assertion, SKIPPED when the binary had no embedded SHA "(likely crates.io install)". ~/.local/bin/apr reports exactly apr 0.60.0 (v0.60.0+no-git).

So the single artifact most likely to be executed was also the one that made the only freshness gate excuse itself. A dogfood run is worthless if it exercised a binary other than the one being shipped.

Both gates now go through . scripts/apr_bin.sh || exit 1; no-SHA is a FAIL unless DOGFOOD_ALLOW_UNPINNED=1 is set deliberately.

5. CLAUDE.md

The "MANDATORY: use apr tools first" block — the most-followed instruction in the repo — told every agent to run bare apr qa. It now pins Step 0 and uses "$APR" throughout.

Verification

check_apr_bin_pinned, check_sourced_libs_option_neutral, check_pass_grep_anchored all exit 0. bashrs errors across both changed scripts: 8 → 5.

Not fixed here

The stale binaries still exist on the box — deleting a developer's ~/.local/bin/apr is an operator decision, not a repo change. Historical specs under docs/specifications/ still quote the old absolute path as a record of past runs; those are records, not instructions. Both tracked separately.

🤖 Generated with Claude Code

…, or dogfooded

Four `apr` binaries were found coexisting on the dev box:

  ~/.local/bin/apr                              0.60.0 (no SHA)  26 days
  ~/.cargo/bin/apr                              0.62.0 (3418754)  FRESH
  /mnt/nvme-raid0/coverage/aprender/release/apr 0.61.0 (9514558)   1 day
  /mnt/nvme-raid0/targets/aprender/release/apr  0.60.0 (c55a333)   6 days

This class has now caused damage three times: qwen-story-daily validated
24-day-old code while reporting green (#2344); a v0.62.0 release smoke-test
read a five-hour-old binary and reported a meaningless pass; and a real
product regression (#2350) stayed hidden 24 days because the nightly ran a
pre-regression binary. Detection existed and was not enough.

1. RESOLUTION: ask cargo, never search. apr_bin.sh derived $APR from
   $CARGO_HOME/bin then PATH — both of which SEARCH, and a search finds the
   wrong binary eventually. It now asks `cargo metadata` for this checkout's
   target_directory. That matters because `.cargo/config.toml` redirects
   target-dir and is GITIGNORED (.gitignore:55) while its siblings
   .cargo/audit.toml and .cargo/mutants.toml are tracked — the directory looks
   version-controlled, the file that moves every build output is not. Measured:
       main checkout  -> /mnt/nvme-raid0/coverage/aprender
       fresh worktree -> <worktree>/target
   So no absolute path is correct in both. $CARGO_HOME/bin is kept LAST (cargo
   install leaves nothing in target/), still freshness-checked.

2. FAIL CLOSED: `APR_BIN_STRICT=1` makes "not a git checkout" a refusal. It
   previously returned 0 with "freshness not asserted" — a fail-OPEN hole in a
   script whose only job is refusing unproven binaries.

3. HARDCODED ABSOLUTE PATHS are now a build failure. check_apr_bin_pinned.sh
   treated `/mnt/.../target/release/apr` as PINNED because it ends in
   target/release/apr — so the exact path that was 6 days and two minor
   versions stale passed the guard. New ABS-APR class, mutation-verified by
   injecting that literal path into qwen-story.sh: RED exit 1, GREEN exit 0
   after revert. The regex is anchored at the path START; the first draft
   matched the `/apr` inside RELATIVE `target/release/apr` and flagged correct
   code. Verified against a 12-case table (4 must match, 8 must not) — this
   regex class has now been gotten wrong four times here, so the table ships
   with it.

4. THE DOGFOOD PROTOCOL certified releases with an unpinned binary:
   - Gate 1 ran `cargo install --path` (writes $CARGO_HOME/bin) and then a
     BARE `apr`, which PATH resolved to the 26-day-old ~/.local/bin copy.
   - Gate 13, the protocol's ONLY freshness assertion, SKIPPED when the binary
     had no embedded SHA "(likely crates.io install)". `~/.local/bin/apr`
     reports exactly `apr 0.60.0 (v0.60.0+no-git)`. The one artifact most
     likely to be executed was the one that made the only gate excuse itself.
   Both now go through `. scripts/apr_bin.sh || exit 1`; no-SHA is a FAIL
   unless DOGFOOD_ALLOW_UNPINNED=1 is set deliberately.

5. CLAUDE.md's "MANDATORY: use apr tools first" block — the most-followed
   instruction in the repo — told every agent to run bare `apr qa`. It now
   pins Step 0 and uses "$APR" throughout.

Guards green: check_apr_bin_pinned, check_sourced_libs_option_neutral,
check_pass_grep_anchored. bashrs errors across both scripts: 8 -> 5.

NOT fixed here: the stale binaries themselves still exist on the box (deleting
another user's ~/.local/bin is an operator decision, not a repo change), and
historical specs under docs/specifications/ still quote the old absolute path
as a record of past runs. Both tracked separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant