perf(ci): 67-E3 — guard-cargo and guard-tree under the 20-minute budget: 7 nightly-class steps move to guards-nightly.yml, guard_tree.sh dispatches in parallel (289 s → 96 s) (PMAT-1098) - #3094
Open
noahgift wants to merge 24 commits into
Conversation
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod
…ier does not do yet The quick tier turns 41 tree-reader targets into a 26-invocation `&&` chain (one cargo per crate, serial, 26 compiles of the shared deps): 55 min on a one-file YAML PR, timed out at 60 under fleet load (#3070). The replacement is ONE build graph filtered by a nextest filterset, so the translation from the registry tokens to that expression is the new load-bearing step. These rows fail now (--filterset is not a flag yet) and pin BOTH polarities: each recognised token becomes exactly one clause, and an unrecognised token is ENV (exit 2) rather than a silently dropped target — a dropped token is a tree-reader test that stops running while the step stays green, which is the failure mode the registry exists to prevent. Row 22 counts clauses against registry lines so nothing can be dropped in bulk either. Row 21 was written fail-open first (`grep -q ":--" && LEFTOVER || NONE-LEFT` passes on EMPTY output) and is hardened here before it was ever green. Refs #3084 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GREEN for the rows added in the previous commit.
`scripts/ci_test_tier.sh --filterset` turns the tree-reader registry's tokens
into a cargo-nextest filterset — `(package(c) & kind(lib))`, `(package(c) &
kind(bin))`, `binary_id(c::name)`, UNIONed with `|`. ci.yml's part-2 quick-tier
step now runs ONE `cargo nextest run --profile ci --workspace --lib --tests
--exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute
-E "$EXPR"` instead of a `&&` chain of one cargo per crate.
The binary-id forms are nextest's own and were VERIFIED on this workspace with
cargo-nextest 0.9.132 (`cargo nextest list --message-format json`), not assumed:
a lib suite's id is the bare package name, an integration target's is
`package::target`, a bin's is `package::bin/name`.
MEASURED on lambda, warm target dir (both listings from the same tree):
one graph 65263 tests, 41 binaries, 39 suites with >=1 test
build+list 96s, execution 220.2s, 316s wall, rc=0, 65263 passed
20-crate 65204 tests, 40 suites
chain (20 distinct crates, not 26 — the 41 tokens group into 20)
The two sets are NOT identical and the difference is entirely FEATURE
UNIFICATION, never target selection:
* A\B = 258 tests. Features the workspace resolve unifies ON and a
per-package resolve leaves OFF (aprender-core hf_hub / inspect::safetensors
/ format::homomorphic / format::quantize, aprender-orchestrate mcp_json,
aprender-test-lib async driver).
* B\A = 199 tests, and ZERO of them exist under the workspace resolve at all —
they are cfg(feature)-gated out. 197 are aprender-test-lib's browser mock
suites, 2 are aprender-present-cli::gate_can_fail. So the FULL tier, which
is `--workspace --lib`, never ran them either: the old chain was compiling a
DIFFERENT aprender-test-lib than the tier it is supposed to approximate.
Aligning the quick tier with the full tier's resolve is the fix, not a loss.
* Suite selection is otherwise exact: 41 clauses -> nextest reports "Starting
65263 tests across 41 binaries". The two suites with no tests are
aprender-core::setfit_conformance and ::falsification_spec_v10_tests, built
and selected in BOTH but empty under default features (they are the full
tier's --features setfit targets).
* `--bins` maps to `kind(bin)`, not to the whole package. The old chain ran
`cargo nextest run -p aprender-present-cli` with NO flags, so it also swept
up `gate_can_fail` — an integration target that is in neither the registry
nor the unwired ledger, i.e. not a tree reader at all. `kind(bin)` selects
exactly what the token claims: the cfg(test) unit tests inside the bins of a
crate that has no src/lib.rs (which is the only reason the token exists —
`--lib` there is `error: no library targets found`).
Both quick-tier steps drop to `timeout-minutes: 20`. 60 is the number that let
this step burn 55 minutes on a one-file YAML PR (run 34449608126) and then die
at the cap under fleet load on #3063 (#3070); the budget is the assertion.
check_tree_reader_tests.sh is untouched and still derives the same 41 targets:
its `full_tier_excludes` regex requires `--workspace --lib` followed IMMEDIATELY
by ` --exclude`, and the new line has `--tests` in between, so it still reads the
full tier's line and only that one (verified by running the regex).
Refs #3084
Closes #3070
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s concurrently, deterministically and fail-closed Row (e) is the discriminator and it reads no clock: two planted guards rendezvous through marker files, so a serial dispatcher fails because the peer never started. Rows (a)-(d),(f) pin the universe, the named failure, the count, byte-identical ordering across two runs, and a guard that dies without a verdict counting as FAILED. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… judge in 20 Measured on run 34449608126 (PR #3074): guard-cargo 43.8 min over 80 steps, guard-tree 27.3 min over 45, and 18.4 of guard-tree's minutes were ONE step running ~35 independent guards one at a time on a 48-core box. That step now dispatches through an xargs pool (bounded -P 8, the clean-room pool is 16 runners on one host): 289 s -> 96 s locally on the real universe, same 41 checks. Seven tree-universe steps (26.8 min: the tier case table, model-tests, book examples, aprender-profile, the fd-0 scan, publish safety, wasm32) move to guards-nightly.yml with their name, run block and docker/mount shape verbatim, so check_guards_are_wired.sh -- which greps every workflow file, now pinned by rows 5-7 of its own case table -- still finds them wired. Both jobs take timeout-minutes: 20, the operator's budget, which is SMALLER than BSE-05's answer and overrides it. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dger, not a ratchet — the one line 67-E3's worker could not write (out of its scope) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
noahgift
enabled auto-merge
September 10, 2026 13:09
…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…rst 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…asset-target-mountpoint
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E3-guards-under-budget # Conflicts: # .github/workflows/ci.yml
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E3-guards-under-budget
…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml
…pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8-67-E3-guards-under-budget # Conflicts: # .github/workflows/ci.yml
…nd give guard-cargo's remaining cargo steps the job's own CARGO_HOME — moving model-tests to the nightly had left guard-cargo with none, and guard_tree_test's case table requires it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er script from main ci.yml: workspace-test keeps main's measured 45-min timeout; sccache path from main; guard-tree keeps BOTH case-table steps (67-E3 dispatcher table + fleet hygiene tables); the BSE-03 SATD ratchet STAYS in guard-cargo (its universe is the diff vs the comparand; nightly HEAD==origin/main makes it vacuous there) and only 'Book rust examples compile' moves; guard-tree/guard-cargo job timeouts 20->30 (fleet-ledger p90 25/45 minus the moved ~23 min, 1.3x margin). scripts/ci_test_tier.sh: taken from main — the branch's copy predates PMAT-3119/3120 and adds nothing E3-specific (12 re-adds of older lines, 229 removals). Pmat-Ticket: PMAT-1098
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row 67-E3 — spec
docs/specifications/06x-release-schedule.md§2 E · epic #3078 · ticket PMAT-1098 · stacked on #3089 (P0-3)Measured (run 34449608126, PR #3074, per-step timestamps): guard-cargo 43.8 min over 80 steps, 13 steps over 60 s; guard-tree 27.3 min, of which 1105 s is the serial
guard_tree.sh --no-cargodispatcher and 404 s the perf-gate case table..github/workflows/guards-nightly.yml(daily + dispatch, clean-room pool, never hosted): seven tree-universe steps moved verbatim (name, run, docker/mounts) — the BSE-17 tier case table (586 s), model-tests falsification suites (256 s), book examples (206 s), aprender-profile lib tests (200 s), the fd-0 inherited-stdin guard (135 s), publish safety (130 s), wasm32 build (94 s) = 26.8 min off the PR path. Ledgerscripts/guards_nightly_manifest.txt; the workflow's first step refuses a manifest entry that is not a verbatim step (vacuity floor 5). Every--self-testcase table of a guard that stayed, every diff-comparand ratchet, and the perf-gate case table stay at PR time.scripts/check_guards_are_wired.shnow scans every workflow file (a guard wired only in guards-nightly.yml counts; wired nowhere is still RED; deleting the nightly brings its guard back as unwired — 3 new self-test rows).scripts/guard_tree.sh --no-cargoin parallel (xargs -P 8,GUARD_TREE_JOBSoverride; bounded at 8 because the clean-room pool is 16 runners on one host): per-guard output captured and printed in deterministic order, same rows andN checks, M failedsummary, fail-closed (a guard that dies without a verdict counts as FAILED).scripts/tests/guard_tree_parallel_test.sh6 rows incl. a marker-file rendezvous that is RED against the serial dispatcher. On lambda: 289 s → 96 s, same 41 checks.timeout-minutes: 20on both jobs. Projection from the one measured run: guard-cargo ≈ 17 min, guard-tree ≤ 17 min — arithmetic, not a p99; the spec's "≤ 20 on three consecutive PR runs" is closed only by the runs of this branch, and the remedy for an overrun is moving the next tree-universe step to the nightly, never widening the timeout.scripts/check_baseline_ratchets.shclassifies the new manifest as a ledger (exact-matched against the workflow), not a ratchet.Orchestrator re-runs: YAML parses, guards-wired / path-filters / runner-labels PASS, parallel case table 6/6,
guard_tree.sh --no-cargo41/0 in 98 s,make gate41/0 in 175 s.🤖 Generated with Claude Code