fix(bench-gpu): make the toolchain gate able to fail, and say why - #940
Merged
MauroToscano merged 2 commits intoAug 18, 2026
Merged
Conversation
Follow-ups from review of the provisioning hardening. - The sanity gate could not fail on a compiler failure. Under `set -e` a non-final operand of an `&&` list is exempt from errexit, and the list's non-zero status does not re-trigger it, so a dead cc/rustc was swallowed and the remote exit status was that of the trailing `rm -rf`. The gate returned 0 and printed "toolchain sane" on a host whose compiler had just crashed. Measured, before -> after: cc SIGSEGV 0 -> 139, cc missing 0 -> 127, cc error 0 -> 1, rustc SIGSEGV 0 -> 139, rustc missing 0 -> 127, healthy 0 -> 0. Every command is now a bare statement; a trap keeps the tmpdir cleanup on both paths. - Distinguish ssh's own exit 255 from a verdict on the toolchain, so a network blip no longer reports the host's compilers as broken. - Run the probe from the repo so rustup resolves the pinned toolchain in rust-toolchain.toml rather than whatever default the image carries. - A failure in this step posted "Run failed" above an EMPTY code block: the PR-comment step tails $RUNNER_TEMP/abba_out.txt, and only the bench step ever wrote it. Record the reason and the compiler output there. - Reword the gate's error. It establishes "cc or rustc could not compile and run a trivial program"; bad RAM is named as one possible cause rather than asserted as the diagnosis. Comments, each previously at odds with the code or with each other: - the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory pressure for the same symptom. The latter now describes OOM as it actually presents (SIGKILL, or an allocation failure) and names jemalloc-sys's CARGO_MAKEFLAGS forwarding, which is what makes the cap bind its nested make. - drop the unmeasured "~10 min dual build", and annotate the 3 min 56 s ETA reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise. - the no-offer error and the env header now list reliability, gpu_frac and cuda_max_good, which they had drifted from. - state the gate's scope: it does not exercise /opt/lambda-vm-sysroot, and a 1 s compile surfaces marginal RAM only sometimes.
Both host-fault messages said "Re-run /bench-gpu to reroll the box", but offer selection is deterministic — `sort_by(.dph_total) | reverse | .[0]` with no machine_id exclusion — so an immediate re-run can re-pick the same machine once it relists and fail identically. Say to wait a few minutes instead, and say why, so the advice matches what the picker actually does. The ssh-255 message is left as an immediate retry: a transport failure is not a verdict on the host, so there is nothing to roll off. Still not an automated reroll (the sibling gpu-tests.yml carries a TRIED machine_id list for that); this only stops the message promising something the selection logic does not do.
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Aug 18, 2026
* ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes The GPU ABBA bench kept failing on rented Vast boxes in ways that looked like code bugs but were the harness building before the box was ready: - The provisioning-complete check fell back to "these few artifacts exist" and started the build while onstart was still populating the sysroot, so the C compiler read a half-written header (truncated bits/timex.h -> "unterminated #ifndef"). Require the "=== done ===" marker only; drop the premature fallback. - Add a toolchain sanity gate (trivial gcc + rustc compile) after provisioning: a bad-RAM host that SIGSEGVs the compiler on the first heavy crate (jemalloc, serde_derive) now fails fast here with a clear message instead of mid-build with an internal-compiler-error backtrace. - Cap the dual build at CARGO_BUILD_JOBS=8 so the initial ramp (LLVM codegen units + jemalloc's nested make -j) can't transiently exceed the box's RAM and trigger OOM-induced compiler crashes. - Filter offers by reliability>=0.95 to skip chronically-flaky hosts before renting (fails safe: over-strict just yields no offers). A full box-reroll (rent another host on a build/prove failure) is the next step but needs a live run to validate against paid infra, so it is left out of this change. * fix(bench-gpu): make the toolchain gate able to fail, and say why (#940) * fix(bench-gpu): make the toolchain gate able to fail, and say why Follow-ups from review of the provisioning hardening. - The sanity gate could not fail on a compiler failure. Under `set -e` a non-final operand of an `&&` list is exempt from errexit, and the list's non-zero status does not re-trigger it, so a dead cc/rustc was swallowed and the remote exit status was that of the trailing `rm -rf`. The gate returned 0 and printed "toolchain sane" on a host whose compiler had just crashed. Measured, before -> after: cc SIGSEGV 0 -> 139, cc missing 0 -> 127, cc error 0 -> 1, rustc SIGSEGV 0 -> 139, rustc missing 0 -> 127, healthy 0 -> 0. Every command is now a bare statement; a trap keeps the tmpdir cleanup on both paths. - Distinguish ssh's own exit 255 from a verdict on the toolchain, so a network blip no longer reports the host's compilers as broken. - Run the probe from the repo so rustup resolves the pinned toolchain in rust-toolchain.toml rather than whatever default the image carries. - A failure in this step posted "Run failed" above an EMPTY code block: the PR-comment step tails $RUNNER_TEMP/abba_out.txt, and only the bench step ever wrote it. Record the reason and the compiler output there. - Reword the gate's error. It establishes "cc or rustc could not compile and run a trivial program"; bad RAM is named as one possible cause rather than asserted as the diagnosis. Comments, each previously at odds with the code or with each other: - the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory pressure for the same symptom. The latter now describes OOM as it actually presents (SIGKILL, or an allocation failure) and names jemalloc-sys's CARGO_MAKEFLAGS forwarding, which is what makes the cap bind its nested make. - drop the unmeasured "~10 min dual build", and annotate the 3 min 56 s ETA reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise. - the no-offer error and the env header now list reliability, gpu_frac and cuda_max_good, which they had drifted from. - state the gate's scope: it does not exercise /opt/lambda-vm-sysroot, and a 1 s compile surfaces marginal RAM only sometimes. * fix(bench-gpu): tell the operator to wait before re-rolling the box Both host-fault messages said "Re-run /bench-gpu to reroll the box", but offer selection is deterministic — `sort_by(.dph_total) | reverse | .[0]` with no machine_id exclusion — so an immediate re-run can re-pick the same machine once it relists and fail identically. Say to wait a few minutes instead, and say why, so the advice matches what the picker actually does. The ssh-255 message is left as an immediate retry: a transport failure is not a verdict on the host, so there is nothing to roll off. Still not an automated reroll (the sibling gpu-tests.yml carries a TRIED machine_id list for that); this only stops the message promising something the selection logic does not do. --------- Co-authored-by: Mauro Toscano <12560266+MauroToscano@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review follow-ups for #939, targeted at that PR's branch so they land with it.
The headline one: the toolchain sanity gate could never fail. Under
set -e, a non-final operand of an&&list is exempt from errexit and the list's non-zero status does not re-trigger it, so a deadcc/rustcwas swallowed and the remote exit status became that of the trailingrm -rf "$d". The step printedtoolchain saneon a host whose compiler had just crashed, then the dual build died ~10 min later with exactly the internal-compiler-error backtrace the gate was added to prevent.Measured against the gate body extracted verbatim from the workflow:
ccexits 139 (ICE/SIGSEGV-like)cckilled by real SIGSEGVccabsent (127)ccexits 1rustcexits 139rustcabsent (127)Worth knowing what the gate did still catch before this change, since it was not fully inert:
mktempfailure, a read-only fs, "compiler exits 0 but emits no/unrunnable binary", and ssh transport failure. What it was blind to was every compiler-side failure — the one class its own comment named.Also in here
rust-toolchain.tomltoolchain instead of the image's default.$RUNNER_TEMP/abba_out.txt, which only the bench step writes, so a pre-bench failure rendered❌ Run failed. Last log lines:above an empty fence — the actionable text never left the Actions log. Failures now record their reason and the compiler output there. (Pre-existing hole; ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes #939 added two new paths into it.)sort_by(.dph_total) | reverse | .[0], nomachine_idexclusion), so an immediate re-run can re-pick the same machine once it relists. They now say to wait a few minutes first, and why. The ssh-255 message still says retry now — a transport failure is not a verdict on the host, so there is nothing to roll off. This is not an automated reroll;gpu-tests.ymlcarries aTRIEDmachine_id list for that, and ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes #939 scopes that work out.Comments
Each of these was at odds with the code or with another comment:
CARGO_BUILD_JOBScomment blamed memory pressure for the same symptom. The latter now describes OOM as it actually presents (SIGKILL, or an allocation failure) and names jemalloc-sys'sCARGO_MAKEFLAGSforwarding — which is the mechanism that makes the cap bind its nestedmake, verified intikv-jemalloc-sysbuild.rs.CARGO_BUILD_JOBS=8will raise.envheader now listreliability,gpu_fracandcuda_max_good, which they had drifted from./opt/lambda-vm-sysroot, and a 1 s compile surfaces marginal RAM only sometimes. It catches a missing or half-installed toolchain every time, which is the real win.Verification
actionlint(with shellcheck) clean;bash -nclean on all 10runblocks; the exit-code matrix above and the tmpdir cleanup were both run against the extracted gate body, and the PR-comment rendering was checked through the workflow's own JS.Not verified — needs a Vast API key and a live box: whether
reliability>=0.95leaves a non-empty pool, and the real-j8build wall. Note also that/bench-gpucannot exercise any of this pre-merge, sinceissue_commentloads the workflow definition from the default branch.