ci(release): P0-1 — four apr binaries on every tag ({cuda,cpu} × {x86_64,aarch64}), verify-apr-assets requires all four, check_release_assets.sh + C13 + a post-publish dogfood row (67-A1, #3082, PMAT-1098) - #3092
Open
noahgift wants to merge 34 commits into
Conversation
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod
…RED at row 0 Ten rows, both polarities: a scratch PATH with every default tool is GREEN and the same PATH minus jq is RED naming MISSING jq; the JSON side-channel parses and records the missing tool; a usage error is exit 2 and a missing tool is exit 1; and the wiring itself is checked — every self-hosted job in fleet-toolset.yml, binary-release.yml and cuda-nightly.yml must run the preflight immediately after checkout, mutation-verified by deleting the step from a workflow copy. Run 34448908554 built a 21 MB CUDA asset on gx10 and died on the upload with `gh: command not found`. Nothing checked the toolset a workflow assumes. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…build scripts/ci_self_hosted_preflight.sh — defaults jq curl git python3 tar sha256sum rustup cargo; --cuda adds nvidia-smi AND asserts a device is actually visible; --need takes the extras a job knows it wants. `gh` is deliberately NOT a default: the CUDA lane moved to REST with curl+python3 because fleet boxes have no gh (#3074), and defaulting it would re-assert the assumption that failed. Exit 1 is a finding about the BOX, exit 2 a finding about the STEP that called this — merged codes send the wrong person. Identity (runner/labels/arch/glibc, driver under --cuda) is printed and, when PREFLIGHT_OUT or RUNNER_TEMP is set, written as JSON: that file is the fleet probe output. 13-row hermetic case table, both polarities, every tool a stub in a scratch PATH so the verdict does not depend on this box. Two rows were RED for the wrong reason first and are recorded as comments: the script called dirname/basename (now parameter expansion) and the nvidia-smi stub catted a fixture, so --cuda reported zero devices because `cat` was not on the scratch PATH. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…spends an hour
The preflight is now the step immediately after checkout in all seven
self-hosted jobs: binary-release.yml build-apr-cuda (--cuda --need docker
objdump), smoke-cuda (same; both GPU boxes carry `docker` as a runner LABEL, so
a pool that stopped matching its own labels is worth one second), and the two
clean-room jobs verify-cuda-assets and summary, which had no checkout at all;
plus the gx10 lane in cuda-nightly.yml. Not before `decide` and not before the
checkout there: on a yield night the box has no working tree, and a step that
bashes a repo path would exit 127 and turn a deliberately-yielded night RED.
fleet-toolset.yml probes all three pools daily at 04:47 UTC and uploads
toolset-<label>/preflight.json, 90-day retention, plus one table in the run
summary. ARTIFACT + SUMMARY, NOT A PR: a PR per day against evidence/ is ~365
machine-written PRs a year through a queue that runs at ~1 PR/hour, so the cost
of the record would exceed the record. A zero-row roll-up exits 1 — no
preflight.json downloaded is a broken probe, not a clean fleet.
The falsifier built its universe from the wrong side and said so: `runs-on`
alone saw 3 jobs, missing build-apr-cuda and smoke-cuda, whose selector is
`${{ fromJSON(matrix.labels) }}` — the two GPU jobs this row exists for. It now
reads the strategy block too, and sees 7.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…0 is missing four Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d smoke it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…h polarities in the table Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… SEC011) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ur apr binaries Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
noahgift
enabled auto-merge
September 10, 2026 12:58
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…apr-assets # Conflicts: # .github/workflows/binary-release.yml
…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 tasks
…rst 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…asset-target-mountpoint
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-A1-four-apr-assets
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-A1-four-apr-assets
…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml
…pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8-67-A1-four-apr-assets
…ls (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e the cuda lane, and smoke-cpu GATES their glibc floor at GLIBC_2.31 — the floor variable it read was defined nowhere check_workflow_env_defined (guard-cargo) failed #3092: smoke-cpu interpolated APR_ASSET_GLIBC_FLOOR, which no job defines, so the floor check recorded a number and gated nothing. It could not honestly be set while build-apr-cpu built NATIVELY: the asset's floor was whatever the build box ran (0.66.0's native x86_64 asset needed GLIBC_2.43 and did not start on 22.04). build-apr-cpu now uses the cuda lane's bullseye sibling-container step verbatim minus --features cuda and the device lines (own cache dir, pre-created target mountpoint), and smoke-cpu declares the floor, so an asset importing a newer GLIBC_* symbol fails the release. Guards: check_workflow_env_defined OK (15 workflows), guard_tree_test 23/0, check_lockfile_current PASS. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r left in binary-release.yml (#3073) binary-release.yml fires on `release: published`, and its pv job ran on ubuntu-latest (aarch64 via cross). Under the operator rule of 2026-09-10 ("WE DO NOT USE HOSTED GITHUB RUNNERS") a tag cut that way is not done. Each pv target now builds natively on the box of its architecture, x86_64 on yoga and aarch64 on gx10, inside rust:1.93.0-bullseye: the sibling-container pattern build-apr-cpu already uses. gnu assets carry a GLIBC_2.31 floor (cross's aarch64 image was 2.31 too; x86_64 came from ubuntu-latest's 2.39, so its floor drops), and musl assets are static, with musl-gcc named as their C compiler. New checks that can fail: the asset runs on its builder, a musl asset has no NEEDED entry, and a gnu asset imports nothing above GLIBC_2.31. Upload is the apr lanes' REST path, since these boxes carry no gh. Asset names are unchanged, so verify-apr-assets' eight pv names still hold. CROSS_VERSION, used only by the old pv job, is gone, and the header describes the new build. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…'s Release file has expired Rehearsal run 34543016302 (v0.67.0-rc.0): both gnu pv lanes built, both musl lanes died at `apt-get update` inside rust:1.93.0-bullseye, exit 100: "Release file for .../bullseye-security/ InRelease is expired". Only the musl lanes run apt (for musl-tools), which is why the gnu lanes on the same two boxes were green. The matrix now names each target's image: gnu stays on rust:1.93.0-bullseye (glibc 2.31 is its floor), musl moves to rust:1.93.0-bookworm, where apt still installs musl-tools. A static musl binary carries no builder glibc, so the floor promise is unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… guard that refuses one (#3073) Operator rule (2026-09-10, verbatim): "WE DO NOT USE HOSTED GITHUB RUNNERS. WE USE GX10 AND LAMBDA-LABS OR YOGA". Three workflows still ran on GitHub-hosted images: - nightly.yml: ubuntu-latest, ubuntu-24.04-arm, macos-latest x2, windows-latest. The Linux builds now run on the fleet (x86_64 on yoga, aarch64 natively on gx10) inside rust:1.93.0-bullseye, so the nightly's glibc floor drops from 2.39 to 2.31. check-activity and the release job run on the clean-room pool, and the release job deletes the old nightly over REST, because the fleet has no gh. No macOS or Windows self-hosted runner is registered, so those targets leave the nightly until one is. - book.yml (build + Pages deploy) and book-contracts.yml (six jobs) move to the clean-room pool. Both are path-filtered (0 and 1 runs in the last 24 h), so the pool barely notices. scripts/check_no_hosted_runners.sh refuses a hosted image token on any non-comment workflow line, not only on `runs-on:` lines: nightly.yml hid `ubuntu-latest` in a matrix value, which a runs-on-only scan never reads (and check_runner_labels.sh exempts hosted jobs by design). A shrink-only `file:job` baseline can excuse a job while its migration is in flight, and a stale entry is RED. The case table has 12 rows: both polarities, a stale-baseline row and an ENV row. Mutation: re-inserting `runs-on: ubuntu-latest` into nightly.yml turns the real-tree run RED (rc=1); restoring it turns it green. The guard is wired through guard_tree.sh's dispatch (check_guards_are_wired.sh PASS), and bashrs reports 0 errors. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n the fleet With the pv lane on gx10 + yoga (this branch), no workflow names a GitHub-hosted image, so the baseline check_no_hosted_runners.sh reads carries no entry. From here a hosted image anywhere is RED. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Folded the pv lane and the rest of #3073 into this branch. It is now one PR that takes every workflow off GitHub-hosted runners:
Release-pipeline rehearsal: Closes #3073 on merge. 🤖 Generated with Claude Code |
Contributor
Author
|
Release-pipeline rehearsal: green end to end.
The earlier rehearsal, rc.0 (run 34543016302), found the musl apt defect that 🤖 Generated with Claude Code |
…ivate root, not the shared home Moving book.yml onto the clean-room pool (#3073) put its three `cargo install` steps (bashrs, the local apr-cli, pmat) under check_cargo_install_private_root.sh, which guard-cargo ran red on yoga-eph (run 34544213071): a self-hosted `cargo install` into the shared ~/.cargo/bin replaces a binary another job on the same box may be exec'ing (aprender#2353). Each install now passes `--root "$RUNNER_TEMP/cargo-tools"`, and that root's bin is added to PATH last, so it resolves first. mdBook, which the guard does not cover, had the same race in ~/bin; it now unpacks into $RUNNER_TEMP/bin. Checked locally: check_cargo_install_private_root.sh OK (43 jobs, 9 installs, none shared) and its case table; check_no_hosted_runners, check_runner_labels, check_workflow_env_defined, check_workflow_path_filters and check_apr_bin_pinned all green. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fied baseline file is RED
check_baseline_ratchets.sh refuses any baseline file it cannot classify as a ratchet or as a stated
exemption ("no rule is how a baseline arrives that nothing ever compares"). It turned guard-tree red on
this PR for scripts/hosted_runner_baseline.txt, which had held nothing since the pv lane moved to the
fleet. check_no_hosted_runners.sh already treats a missing baseline as zero exemptions, the strictest
state, so the file goes. A later migration that needs an entry must add the file and classify it in the
same change. Checked locally: check_baseline_ratchets, check_no_hosted_runners (real tree + 12-row case
table) and check_guards_are_wired are all green.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row 67-A1 (P0-1, #3082) — spec
docs/specifications/06x-release-schedule.md§2 A · epic #3078 · ticket PMAT-1098 · stacked on #3088 (67-C2)v0.66.0 was cut with zero
aprbinaries because no gate asserted the asset set. Live today (bash scripts/check_release_assets.sh v0.66.0): the two CUDA tarballs readok, the two-cputarballs and their checksums areMISSING— exit 1. That is the P0 made visible by a command.build-apr-cpuinbinary-release.yml: same two disposable runners as the CUDA lane (gx10-eph aarch64, yoga-eph x86_64; labels…, ephemeral, docker), preflight first, DEFAULT features (crates/apr-cli/Cargo.toml:defaultcarries nocuda;cudais opt-in), assetapr-<tag>-<target>-cpu.tar.gz+.sha256, glibc floor recorded; the ci(release): CUDA apr binaries for x86_64 and aarch64 on every release — a hard requirement, verified in the bytes and on both GPU hosts (PMAT-1096, #2869) #3072 discriminator inverted —grep -ac libcuda.so aprmust be 0 or the job fails.verify-cuda-assets→verify-apr-assets: all four apr tarballs + four.sha256+ the eightpvassets, by callingscripts/check_release_assets.sh <tag>(0 complete · 1 missing-by-name · 2 ENV;ghwhen present else REST with the job token;--assets-from/RELEASE_ASSETS_FIXTUREoffline seam; 11-row--selftestincl. the registered mutation: remove the aarch64 cpu asset → RED).scripts/tests/check_release_assets_test.sh13 rows.smoke-cpu: the x86_64 CPU asset on a GPU-less clean-room runner and each arch's CPU asset on its GPU host — checksum,apr --versioncarries the tag,libcuda.socount 0, glibc floor fromobjdump -T.scripts/release_criteria.shC13 reads the tag from the root manifest and runs the checker; three new self-test rows (GREEN/RED/ENV).scripts/dogfood.shgainsrelease-assets: PASS/FAIL post-publish, SKIP-with-reason pre-publish (assets exist only after the tag;PREPUBLISH_DEFERRABLEis outside this row's scope, growing it is a follow-up).Orchestrator re-runs:
--selftest11/11,release_criteria.sh --self-test10/10, YAML parses, runner-label / path-filter / guards-wired PASS, bashrs 0 gating, live v0.66.0 check exit 1 naming the two-cpuassets,make gate41 checks 0 failed.Note: this branch predates #3086 (bullseye container builds);
build-apr-cpumirrors the lane's shape at branch time. The merge reconciles the two lanes; the smoke's floor assertion arms only whenAPR_ASSET_GLIBC_FLOORis set, which #3086's containerised lane should export.🤖 Generated with Claude Code
Closes #3073.