feat(crosscheck): carry the Pi reviewer closure in the Azure model image by ruby-dlee · Pull Request #246 · ruby-dlee/firstmate

ruby-dlee · 2026-08-18T18:50:15Z

What was broken

The Azure Crosscheck model guest has had a working pi branch since early August, bin/fm-crosscheck-azure.py admits pi reviewers (provider host, auth.json credential name, PI_CODING_AGENT_DIR), and docs/azure-crosscheck.md:62 already told operators the image pins "the supported Codex, Claude, or Pi reviewer CLI".

The tracked build contract installed only the Codex and Claude CLIs. So a Pi reviewer reached a live model VM, ran, and died on pi: command not found after the VM was already paid for. Documentation described a closure the image did not carry.

What this changes

Adds the Pi closure to docs/azure-crosscheck/model-image.json, pinned exactly the way bin/fm-azure-cell-image.sh pins it for crewmates, and against the same tarball, so a Pi reviewer and a Pi author run one identical agent rather than two versions that could disagree for reasons a review would report as a finding.

Pi is an npm package with a #!/usr/bin/env node entrypoint, not a static binary, so:

its Node runtime is pinned by the same URL/size/SHA-256 contract rather than pulled from a distribution repository mid-build, and the test refuses a deb.nodesource.com / add-apt-repository install;
PATH is exported explicitly, because every other closure check in this build invokes an absolute path and would pass on an image where env node cannot resolve.

Two gaps the previous contract could not catch, closed for all three CLIs:

Version is asserted, not just exercised. pi --version exits 0 for any version, so running it cannot show the digest-pinned tarball is what reached PATH. piVersion is a tracked parameter and the build compares against it.
The closure is re-verified after the purge. Every check ran before the ambient-credential purge, which is a broad find / -exec rm -rf and therefore the one step capable of removing a CLI. The closure is now re-verified as the build's last act.

.pi joins the ambient-credential purge, and npm's build-time state is removed, so the image still ships with no ambient credential.

Test

The declaration test gains an assertion that every closure parameter is referenced by a build step. Parameter presence alone cannot distinguish a closure the image carries from one it merely declares.

Five mutations, each confirmed to change the file before the run:

mutation	result
drop the `piTarballSha256` parameter	red
drop the version assertion	red
replace the pinned runtime with a nodesource install	red
drop `.pi` from the credential purge	red
keep `piTarballSha256` declared, replace its verification step with an `echo`	red (the case parameter-presence cannot catch)

tests/fm-crosscheck-azure.test.sh passes; shellcheck -x on the touched test passes; bin/fm-behavior-shards.sh --check 8 passes.

Scope

Tracked declaration only. image-build, promoting the managed image into the Compute Gallery (the reviewer SKUs need a DiskControllerTypes feature a managed image cannot carry), and repointing model_image_id remain operator lanes. The docs now record that gallery promotion is the one step of this contract the bounded command does not own, which is why a rebuilt image does not reach reviews until it is promoted.

ruby-dlee · 2026-08-18T19:12:45Z

Adversarial review returned DO NOT MERGE. Fixed in d7efbe26.

The important finding: three of my four headline properties were pinned by no test. The new assertion checked that each closure parameter appeared somewhere in the build. That is a proxy, and an echo naming the parameter satisfies it. Five mutations survived green:

mutation	was	now
Pi digest check replaced by an echo that keeps the parameter reference	green	red
both version comparisons downgraded to an echo	green	red
entire post-purge re-verification deleted	green	red
`xz-utils` dropped while the build still runs `tar -xJ`	green	red
`export PATH` deleted	green	red

The xz-utils one is a guaranteed billable bake failure that the suite waved through. The assertions now pin steps rather than mentions: each pinned digest must be consumed by exactly one sha256sum -c step; the Pi version must be consumed by a comparison against its tracked parameter, twice; the closure must be re-verified at an index after the purge; xz-utils must be installed whenever the build untars an xz archive. A sixth mutation removing only the post-purge block while leaving the version comparisons in place is also red.

One claim was false. "Digest-pinned closure" is not true for Pi. npm install -g is a network install: the digest pins the published tarball, and the agent then arrives as ~127 separately fetched packages. Pi's own six sibling packages carry a resolved URL and no integrity hash in its shipped shrinkwrap, so a republished @earendil-works/pi-*@0.84.1 would enter this credentialed image with every digest check passing and pi --version still printing 0.84.1. This is the same exposure the crewmate cell image already accepts for the same package. It is now recorded as what it is.

Also fixed:

The config example still showed the managed-image form of model_image_id, 22 lines below the new paragraph saying it must be a gallery version. An operator following it fails at VM create, since the reviewer SKUs need a DiskControllerTypes feature a managed image cannot carry.
Admission never checks that the configured image carries the harness it admits. The build writes a pi-tarball-sha256 tag and nothing reads it, so this branch alone does not stop the next Pi reviewer failing inside a paid VM. Recorded as an open gap.
The model guest launches pi without --offline against a deny-all egress policy, so its startup update and telemetry calls are dropped and each waits out its timeout on a paid VM. Recorded; the guest is deliberately not changed here, to keep this PR to the image contract.
npm install was the only network step in this build with no timeout: a registry stall burned the full 60-minute build timeout and then billed a failed bake. Bounded at 900s.

Correction to the PR description: it claimed a Pi reviewer "reached a live model VM, ran, and died on pi: command not found after the VM was already paid for." The review checked and no pi reviewer has ever run: the roster is 1 claude + 3 codex and there are zero pi reviews in state. The defect is real and latent, not an incident that happened. I should not have written it as one.

Credential staging is the remaining half of the Pi lane and is out of scope here: create_credential_archive stages the whole pooled auth.json, so eight OAuth profiles would reach one reviewer VM and only the first is readable. #248 addresses that.

The Azure Crosscheck model guest has had a working `pi` branch and the adapter has admitted `pi` reviewers for weeks, and docs/azure-crosscheck.md already claimed the image pins "the supported Codex, Claude, or Pi reviewer CLI". The tracked build contract installed only the Codex and Claude CLIs, so every Pi reviewer that reached a live model VM died on `pi: command not found` after the VM was already paid for. Pin the Pi closure the same way the crewmate cell image does, with the same tarball, so a Pi reviewer and a Pi author run one identical agent. Pi is an npm package with a `#!/usr/bin/env node` entrypoint rather than a static binary, so its Node runtime is pinned by the same URL/size/SHA-256 contract instead of being pulled from a distribution repository mid-build, and PATH is named explicitly because every other closure check in this build runs an absolute path and would pass with a PATH that cannot resolve `node`. Two things the previous contract could not prove, now proven: - `pi --version` exits 0 for any version, so it cannot show that the digest-pinned tarball is what reached PATH. The exact version is a tracked parameter and the build asserts it. - Every closure check ran before the ambient-credential purge, which is a broad `find / -exec rm -rf` and therefore the one step capable of taking a CLI back out. The closure is re-verified as the build's last act. The declaration test gains an assertion that every closure parameter is actually referenced by a build step: parameter presence alone cannot tell a carried closure apart from a declared one. Verification: five mutations applied to the template, each confirmed to change the file before running, each red -- dropped parameter, dropped version assertion, a nodesource install replacing the pinned runtime, the dropped `.pi` purge, and a declared `piTarballSha256` whose verification step was replaced by an echo (the case parameter-presence cannot catch). tests/fm-crosscheck-azure.test.sh and shellcheck pass. This is the tracked declaration only. Building the image, promoting it into the Compute Gallery, and repointing `model_image_id` stay operator lanes.

…erclaiming Pi An adversarial review of this branch found that three of its four headline properties were held by no test, and that one claim was simply false. The new assertion checked that each closure parameter appeared somewhere in the build. "Appears somewhere" is a proxy, and an `echo` naming the parameter satisfies it. Five mutations survived green: the Pi digest check replaced by an echo that kept the reference, both version comparisons downgraded to an echo, the entire post-purge re-verification deleted, `xz-utils` dropped while the build still ran `tar -xJ` (a guaranteed billable bake failure), and the PATH export deleted. The assertions now pin the steps rather than the mentions: each pinned digest must be consumed by exactly one `sha256sum -c` step, the Pi version must be consumed by a comparison against its tracked parameter twice, the closure must be re-verified at an index after the purge, `xz-utils` must be installed whenever the build untars an xz archive, and the build must fix a PATH. All five mutations are red, plus one that removes only the post-purge block while leaving the version comparisons in place. The false claim was "digest-pinned closure". `npm install -g` is a network install: the digest pins the published Pi tarball, and the agent then arrives as roughly 127 separately fetched packages. Pi's own six sibling packages carry a `resolved` URL and no `integrity` hash in its shipped shrinkwrap, so a republished `@earendil-works/pi-*@0.84.1` would enter this credentialed image with every digest check passing and `pi --version` still printing 0.84.1. That is the same exposure the crewmate cell image already accepts for the same package. It is now recorded as what it is instead of described as pinning. Also from the same review: - The config example still showed the managed-image form of `model_image_id`, 22 lines below the new paragraph explaining it must be a gallery version. An operator following the example fails at VM create, because the reviewer SKUs need a DiskControllerTypes feature a managed image cannot carry. - Admission never checks that the configured image carries the harness it admits. The build writes a `pi-tarball-sha256` tag and nothing reads it, so this branch does not by itself stop the next Pi reviewer from failing inside a paid VM. Recorded as an open gap rather than implied to be covered. - The model guest launches `pi` without `--offline` against a deny-all egress policy, so its startup update and telemetry calls are dropped and each waits out its timeout on a paid VM. Recorded; the guest is not changed here. - The npm install was the only network step in this build with no timeout. A registry stall burned the full 60-minute build timeout and then billed a failed bake. Bounded at 900s, matching the curl bounds beside it. Two documented phrasings of the same fact were reconciled.

A verification pass found ten findings against this section, three of them merge-blocking. All ten are addressed here, docs only, one file. The image claim is refuted rather than softened. The current fm-ccm image does carry pi. Config model_image_id names gallery version 1.0.1787092687, whose source managed image img-fm7c799d-ccm-1.0.1787091895 carries the pi-tarball-sha256 and node-tarball-sha256 tags that only a build from the Pi-carrying declaration (#246) can write, its Image Builder run succeeded on 2026-08-18 between 22:26:20Z and 22:36:46Z, after #246 landed on main at 20:51:35Z, and that declaration asserts pi --version against a tracked version twice, before and after the credential purge. The digests match docs/azure-crosscheck/model-image-closure.json for pi-coding-agent 0.84.1 and Node v22.23.2. The old finding was correct about 1.0.1786915905, which the config stopped naming on 2026-08-18, and it was already stale when it was written into docs/azure-crosscheck.md on 2026-08-20. The dangling "ledger M29" provenance is dropped. The operative fact the section omitted is now stated: the lane is off because crosscheck-azure.json has enabled:false. The status line no longer overclaims. BUILT AND SERVING becomes BUILT 2026-08-20 with the fact that matters first: zero GLM reviews have ever completed, six attempts on 2026-08-20 all recorded tool-failure, and the lane's only verdict came from the pi-codex fallback. The current roster state is reflected, including the restored fallback entries with crosscheck-same-model on. Other corrections: 510K tokens was one hour reported as a day, and the measured total is 727,136; the reasoning_effort probe is restated as an unretained observation; the five-tool-turns figure is removed because no run record carries a turn count; both per-minute limits are stated and the token attribution is labelled inference; the account of what killed each attempt is replaced with what the ledger records; the C1 forward reference is dropped; the zeroed GLM cost table is recorded as a spend-visibility gap; the Cost Management implication is dropped while keeping the fact; and three Work items that appeared in neither list are now covered. R9 is left alone: it already says a pi review on the current image is unproven, which stays true and is consistent with the corrected text.

… owed (#275) * docs(azure): make the R6 status true to what landed and what is still owed R6 still read NOT DONE with a work list describing work that merged in #264 and is now serving. Records what landed, what is genuinely still owed (the live end-to-end GLM review, blocked on a 25K token-per-minute Fireworks quota that is an owner portal action; the startup-credit decrement check; the compartment lane's image rebake), and leaves the acceptance sentence unmet because it is. * docs(azure): correct the R6 status to what the artifacts actually show A verification pass found ten findings against this section, three of them merge-blocking. All ten are addressed here, docs only, one file. The image claim is refuted rather than softened. The current fm-ccm image does carry pi. Config model_image_id names gallery version 1.0.1787092687, whose source managed image img-fm7c799d-ccm-1.0.1787091895 carries the pi-tarball-sha256 and node-tarball-sha256 tags that only a build from the Pi-carrying declaration (#246) can write, its Image Builder run succeeded on 2026-08-18 between 22:26:20Z and 22:36:46Z, after #246 landed on main at 20:51:35Z, and that declaration asserts pi --version against a tracked version twice, before and after the credential purge. The digests match docs/azure-crosscheck/model-image-closure.json for pi-coding-agent 0.84.1 and Node v22.23.2. The old finding was correct about 1.0.1786915905, which the config stopped naming on 2026-08-18, and it was already stale when it was written into docs/azure-crosscheck.md on 2026-08-20. The dangling "ledger M29" provenance is dropped. The operative fact the section omitted is now stated: the lane is off because crosscheck-azure.json has enabled:false. The status line no longer overclaims. BUILT AND SERVING becomes BUILT 2026-08-20 with the fact that matters first: zero GLM reviews have ever completed, six attempts on 2026-08-20 all recorded tool-failure, and the lane's only verdict came from the pi-codex fallback. The current roster state is reflected, including the restored fallback entries with crosscheck-same-model on. Other corrections: 510K tokens was one hour reported as a day, and the measured total is 727,136; the reasoning_effort probe is restated as an unretained observation; the five-tool-turns figure is removed because no run record carries a turn count; both per-minute limits are stated and the token attribution is labelled inference; the account of what killed each attempt is replaced with what the ledger records; the C1 forward reference is dropped; the zeroed GLM cost table is recorded as a spend-visibility gap; the Cost Management implication is dropped while keeping the fact; and three Work items that appeared in neither list are now covered. R9 is left alone: it already says a pi review on the current image is unproven, which stays true and is consistent with the corrected text. * docs(azure): keep the quota wording consistent with the inference caveat * docs(azure): retire the fm-ccm no-pi claim at its source and in C1 Scope expansion approved by the coordinator: the requirement doc corrected this claim while the component doc it originated in still asserted it, and C1 (merged in #274 while this branch was open) had since built five further claims on top of it. docs/azure-crosscheck.md, where the sentence originated, now records the corrected limit with the evidence chain: the current fm-ccm image carries pi, and the lane is off because crosscheck-azure.json has enabled:false. The 2026-08-16 measurement is kept and attributed, since M29 is not a dangling reference, only one that lives in the owner's mutation ledger outside this repository, which is now said explicitly. Two neighbouring sentences depended on the false one: the instrumentation follow-up described as bound to an image rebake, and the phases paragraph attributing the lane's silence to a missing image capability. Both now name the flag. C1 depended on it five times and one of those dependencies was independently wrong: it stated the operator home has no config/crosscheck-azure.json. The file exists, carries enabled:false, and names a current model_image_id. Also corrected: the status line and order-of-work entry no longer say the phases wait on a rebake; the argument that the 75 minutes cannot have come from a compartment run after 2026-08-16 is withdrawn rather than restated, since its premise was the false claim; and unmeasurable becomes unmeasured, which is a reversible condition. C1's "known contradiction" paragraph is resolved rather than left stale: R6's status line no longer reads NOT DONE. It now carries the correction that matters to C1, which is that GLM has never completed a review, so the pi-codex fallback is what served the one measured run.

ruby-dlee force-pushed the fm/crosscheck-pi-model-image branch from d7efbe2 to 99f63e4 Compare August 18, 2026 20:01

ruby-dlee added 2 commits August 18, 2026 16:34

ruby-dlee force-pushed the fm/crosscheck-pi-model-image branch from 99f63e4 to 40d98f6 Compare August 18, 2026 20:34

ruby-dlee merged commit e16bf4e into main Aug 18, 2026
13 checks passed

ruby-dlee deleted the fm/crosscheck-pi-model-image branch August 18, 2026 20:51

ruby-dlee mentioned this pull request Aug 18, 2026

feat(crosscheck): track the model-image closure so a built image can be rebuilt #249

Merged

ruby-dlee mentioned this pull request Aug 20, 2026

docs(azure): make the R6 status true to what landed and what is still owed #275

Merged

ruby-dlee mentioned this pull request Aug 20, 2026

feat(crosscheck): refuse a model image that does not attest the dispatched harness #276

Merged

Navigation Menu

feat(crosscheck): carry the Pi reviewer closure in the Azure model image - #246

feat(crosscheck): carry the Pi reviewer closure in the Azure model image#246
ruby-dlee merged 2 commits into
mainfrom
fm/crosscheck-pi-model-image

ruby-dlee commented Aug 18, 2026

Uh oh!

ruby-dlee commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ruby-dlee commented Aug 18, 2026

What was broken

What this changes

Test

Scope

Uh oh!

ruby-dlee commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant