ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes - #939
Conversation
The GPU ABBA bench kept failing on rented Vast boxes in ways that looked like code bugs but were the harness building before the box was ready: - The provisioning-complete check fell back to "these few artifacts exist" and started the build while onstart was still populating the sysroot, so the C compiler read a half-written header (truncated bits/timex.h -> "unterminated #ifndef"). Require the "=== done ===" marker only; drop the premature fallback. - Add a toolchain sanity gate (trivial gcc + rustc compile) after provisioning: a bad-RAM host that SIGSEGVs the compiler on the first heavy crate (jemalloc, serde_derive) now fails fast here with a clear message instead of mid-build with an internal-compiler-error backtrace. - Cap the dual build at CARGO_BUILD_JOBS=8 so the initial ramp (LLVM codegen units + jemalloc's nested make -j) can't transiently exceed the box's RAM and trigger OOM-induced compiler crashes. - Filter offers by reliability>=0.95 to skip chronically-flaky hosts before renting (fails safe: over-strict just yields no offers). A full box-reroll (rent another host on a build/prove failure) is the next step but needs a live run to validate against paid infra, so it is left out of this change.
|
/ai-review |
Codex Code ReviewNo actionable issues found in the PR changes. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Review — CI-only change, no runtime/prover code touchedThe four changes are individually sound and the reasoning in the comments is accurate. Verified against the repo: No Critical/High issues. Three Low ones: 1. The toolchain sanity gate tests a different toolchain than the build. Details inline — short version: the sysroot corruption this PR is chasing is under 2. Worth adding 3. The premature-fallback bug still lives in Nothing blocking. |
AI ReviewPR #939 · 1 changed files Findings
Status column reflects the verdict from the verifier: deepseek-verifier (openrouter/deepseek/deepseek-v4-pro). AI-002: Toolchain sanity check cannot detect a broken compiler (set -e + && + trailing rm masks the exit code)
Claim The new toolchain sanity check almost always returns success (exit 0) even when both cc and rustc are broken (e.g. SIGSEGV from bad RAM), which is the exact failure mode it is meant to catch. The remote script is Evidence Lines 394-399: Suggested fix Drop the Reviewer Lanes
Verification Lanes
Native Codex and Claude reviews run separately and post their own comments. They are not included in this structured provenance report. Discarded candidates (3) — rejected by the verifier
Raw lane outputs, candidates, final issues, and model metrics are uploaded as workflow artifacts. |
* fix(bench-gpu): make the toolchain gate able to fail, and say why Follow-ups from review of the provisioning hardening. - The sanity gate could not fail on a compiler failure. Under `set -e` a non-final operand of an `&&` list is exempt from errexit, and the list's non-zero status does not re-trigger it, so a dead cc/rustc was swallowed and the remote exit status was that of the trailing `rm -rf`. The gate returned 0 and printed "toolchain sane" on a host whose compiler had just crashed. Measured, before -> after: cc SIGSEGV 0 -> 139, cc missing 0 -> 127, cc error 0 -> 1, rustc SIGSEGV 0 -> 139, rustc missing 0 -> 127, healthy 0 -> 0. Every command is now a bare statement; a trap keeps the tmpdir cleanup on both paths. - Distinguish ssh's own exit 255 from a verdict on the toolchain, so a network blip no longer reports the host's compilers as broken. - Run the probe from the repo so rustup resolves the pinned toolchain in rust-toolchain.toml rather than whatever default the image carries. - A failure in this step posted "Run failed" above an EMPTY code block: the PR-comment step tails $RUNNER_TEMP/abba_out.txt, and only the bench step ever wrote it. Record the reason and the compiler output there. - Reword the gate's error. It establishes "cc or rustc could not compile and run a trivial program"; bad RAM is named as one possible cause rather than asserted as the diagnosis. Comments, each previously at odds with the code or with each other: - the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory pressure for the same symptom. The latter now describes OOM as it actually presents (SIGKILL, or an allocation failure) and names jemalloc-sys's CARGO_MAKEFLAGS forwarding, which is what makes the cap bind its nested make. - drop the unmeasured "~10 min dual build", and annotate the 3 min 56 s ETA reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise. - the no-offer error and the env header now list reliability, gpu_frac and cuda_max_good, which they had drifted from. - state the gate's scope: it does not exercise /opt/lambda-vm-sysroot, and a 1 s compile surfaces marginal RAM only sometimes. * fix(bench-gpu): tell the operator to wait before re-rolling the box Both host-fault messages said "Re-run /bench-gpu to reroll the box", but offer selection is deterministic — `sort_by(.dph_total) | reverse | .[0]` with no machine_id exclusion — so an immediate re-run can re-pick the same machine once it relists and fail identically. Say to wait a few minutes instead, and say why, so the advice matches what the picker actually does. The ssh-255 message is left as an immediate retry: a transport failure is not a verdict on the host, so there is nothing to roll off. Still not an automated reroll (the sibling gpu-tests.yml carries a TRIED machine_id list for that); this only stops the message promising something the selection logic does not do.
2 participants
The GPU ABBA bench kept failing on rented Vast boxes in ways that looked like code bugs but were the harness building before the box was ready:
A full box-reroll (rent another host on a build/prove failure) is the next step but needs a live run to validate against paid infra, so it is left out of this change.