Skip to content

fix(bench-gpu): make the toolchain gate able to fail, and say why - #940

Merged
MauroToscano merged 2 commits into
fix/gpu-bench-provisioningfrom
fix/gpu-bench-provisioning-review
Aug 18, 2026
Merged

fix(bench-gpu): make the toolchain gate able to fail, and say why#940
MauroToscano merged 2 commits into
fix/gpu-bench-provisioningfrom
fix/gpu-bench-provisioning-review

Conversation

@MauroToscano

@MauroToscano MauroToscano commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Review follow-ups for #939, targeted at that PR's branch so they land with it.

The headline one: the toolchain sanity gate could never fail. Under set -e, a non-final operand of an && list is exempt from errexit and the list's non-zero status does not re-trigger it, so a dead cc/rustc was swallowed and the remote exit status became that of the trailing rm -rf "$d". The step printed toolchain sane on a host whose compiler had just crashed, then the dual build died ~10 min later with exactly the internal-compiler-error backtrace the gate was added to prevent.

Measured against the gate body extracted verbatim from the workflow:

injected failure before after
cc exits 139 (ICE/SIGSEGV-like) 0 139
cc killed by real SIGSEGV 0 139
cc absent (127) 0 127
cc exits 1 0 1
rustc exits 139 0 139
rustc absent (127) 0 127
healthy box 0 0

Worth knowing what the gate did still catch before this change, since it was not fully inert: mktemp failure, a read-only fs, "compiler exits 0 but emits no/unrunnable binary", and ssh transport failure. What it was blind to was every compiler-side failure — the one class its own comment named.

Also in here

  • ssh exit 255 is no longer read as a toolchain verdict. A network blip previously printed "this host's gcc/rustc is broken".
  • The probe runs from the repo, so rustup resolves the pinned rust-toolchain.toml toolchain instead of the image's default.
  • A failure in this step used to post an empty PR comment. The comment step tails $RUNNER_TEMP/abba_out.txt, which only the bench step writes, so a pre-bench failure rendered ❌ Run failed. Last log lines: above an empty fence — the actionable text never left the Actions log. Failures now record their reason and the compiler output there. (Pre-existing hole; ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes #939 added two new paths into it.)
  • The gate's error no longer asserts "bad RAM." It establishes "cc or rustc could not compile and run a trivial program"; bad RAM is offered as one cause alongside a partially provisioned image, which is the likelier one.
  • The reroll advice now matches what the picker does. Both host-fault messages said "Re-run /bench-gpu to reroll the box", but selection is deterministic (sort_by(.dph_total) | reverse | .[0], no machine_id exclusion), so an immediate re-run can re-pick the same machine once it relists. They now say to wait a few minutes first, and why. The ssh-255 message still says retry now — a transport failure is not a verdict on the host, so there is nothing to roll off. This is not an automated reroll; gpu-tests.yml carries a TRIED machine_id list for that, and ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes #939 scopes that work out.

Comments

Each of these was at odds with the code or with another comment:

  • the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory pressure for the same symptom. The latter now describes OOM as it actually presents (SIGKILL, or an allocation failure) and names jemalloc-sys's CARGO_MAKEFLAGS forwarding — which is the mechanism that makes the cap bind its nested make, verified in tikv-jemalloc-sys build.rs.
  • dropped the unmeasured "~10 min dual build"; annotated the pre-existing "3 min 56 s" ETA reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise.
  • the no-offer error and the env header now list reliability, gpu_frac and cuda_max_good, which they had drifted from.
  • the gate now states its own scope: it does not exercise /opt/lambda-vm-sysroot, and a 1 s compile surfaces marginal RAM only sometimes. It catches a missing or half-installed toolchain every time, which is the real win.

Verification

actionlint (with shellcheck) clean; bash -n clean on all 10 run blocks; the exit-code matrix above and the tmpdir cleanup were both run against the extracted gate body, and the PR-comment rendering was checked through the workflow's own JS.

Not verified — needs a Vast API key and a live box: whether reliability>=0.95 leaves a non-empty pool, and the real -j8 build wall. Note also that /bench-gpu cannot exercise any of this pre-merge, since issue_comment loads the workflow definition from the default branch.

Follow-ups from review of the provisioning hardening.

- The sanity gate could not fail on a compiler failure. Under `set -e` a
  non-final operand of an `&&` list is exempt from errexit, and the list's
  non-zero status does not re-trigger it, so a dead cc/rustc was swallowed
  and the remote exit status was that of the trailing `rm -rf`. The gate
  returned 0 and printed "toolchain sane" on a host whose compiler had just
  crashed. Measured, before -> after: cc SIGSEGV 0 -> 139, cc missing
  0 -> 127, cc error 0 -> 1, rustc SIGSEGV 0 -> 139, rustc missing 0 -> 127,
  healthy 0 -> 0. Every command is now a bare statement; a trap keeps the
  tmpdir cleanup on both paths.
- Distinguish ssh's own exit 255 from a verdict on the toolchain, so a
  network blip no longer reports the host's compilers as broken.
- Run the probe from the repo so rustup resolves the pinned toolchain in
  rust-toolchain.toml rather than whatever default the image carries.
- A failure in this step posted "Run failed" above an EMPTY code block: the
  PR-comment step tails $RUNNER_TEMP/abba_out.txt, and only the bench step
  ever wrote it. Record the reason and the compiler output there.
- Reword the gate's error. It establishes "cc or rustc could not compile and
  run a trivial program"; bad RAM is named as one possible cause rather than
  asserted as the diagnosis.

Comments, each previously at odds with the code or with each other:

- the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory
  pressure for the same symptom. The latter now describes OOM as it actually
  presents (SIGKILL, or an allocation failure) and names jemalloc-sys's
  CARGO_MAKEFLAGS forwarding, which is what makes the cap bind its nested make.
- drop the unmeasured "~10 min dual build", and annotate the 3 min 56 s ETA
  reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise.
- the no-offer error and the env header now list reliability, gpu_frac and
  cuda_max_good, which they had drifted from.
- state the gate's scope: it does not exercise /opt/lambda-vm-sysroot, and a
  1 s compile surfaces marginal RAM only sometimes.
Both host-fault messages said "Re-run /bench-gpu to reroll the box", but
offer selection is deterministic — `sort_by(.dph_total) | reverse | .[0]`
with no machine_id exclusion — so an immediate re-run can re-pick the same
machine once it relists and fail identically. Say to wait a few minutes
instead, and say why, so the advice matches what the picker actually does.

The ssh-255 message is left as an immediate retry: a transport failure is
not a verdict on the host, so there is nothing to roll off.

Still not an automated reroll (the sibling gpu-tests.yml carries a TRIED
machine_id list for that); this only stops the message promising something
the selection logic does not do.
@MauroToscano
MauroToscano merged commit 380450f into fix/gpu-bench-provisioning Aug 18, 2026
8 checks passed
@MauroToscano
MauroToscano deleted the fix/gpu-bench-provisioning-review branch August 18, 2026 22:20
github-merge-queue Bot pushed a commit that referenced this pull request Aug 18, 2026
* ci(bench-gpu): stop building on half-provisioned or bad-RAM boxes

The GPU ABBA bench kept failing on rented Vast boxes in ways that looked like
code bugs but were the harness building before the box was ready:

- The provisioning-complete check fell back to "these few artifacts exist"
  and started the build while onstart was still populating the sysroot, so
  the C compiler read a half-written header (truncated bits/timex.h ->
  "unterminated #ifndef"). Require the "=== done ===" marker only; drop the
  premature fallback.
- Add a toolchain sanity gate (trivial gcc + rustc compile) after
  provisioning: a bad-RAM host that SIGSEGVs the compiler on the first heavy
  crate (jemalloc, serde_derive) now fails fast here with a clear message
  instead of mid-build with an internal-compiler-error backtrace.
- Cap the dual build at CARGO_BUILD_JOBS=8 so the initial ramp (LLVM codegen
  units + jemalloc's nested make -j) can't transiently exceed the box's RAM
  and trigger OOM-induced compiler crashes.
- Filter offers by reliability>=0.95 to skip chronically-flaky hosts before
  renting (fails safe: over-strict just yields no offers).

A full box-reroll (rent another host on a build/prove failure) is the next
step but needs a live run to validate against paid infra, so it is left out
of this change.

* fix(bench-gpu): make the toolchain gate able to fail, and say why (#940)

* fix(bench-gpu): make the toolchain gate able to fail, and say why

Follow-ups from review of the provisioning hardening.

- The sanity gate could not fail on a compiler failure. Under `set -e` a
  non-final operand of an `&&` list is exempt from errexit, and the list's
  non-zero status does not re-trigger it, so a dead cc/rustc was swallowed
  and the remote exit status was that of the trailing `rm -rf`. The gate
  returned 0 and printed "toolchain sane" on a host whose compiler had just
  crashed. Measured, before -> after: cc SIGSEGV 0 -> 139, cc missing
  0 -> 127, cc error 0 -> 1, rustc SIGSEGV 0 -> 139, rustc missing 0 -> 127,
  healthy 0 -> 0. Every command is now a bare statement; a trap keeps the
  tmpdir cleanup on both paths.
- Distinguish ssh's own exit 255 from a verdict on the toolchain, so a
  network blip no longer reports the host's compilers as broken.
- Run the probe from the repo so rustup resolves the pinned toolchain in
  rust-toolchain.toml rather than whatever default the image carries.
- A failure in this step posted "Run failed" above an EMPTY code block: the
  PR-comment step tails $RUNNER_TEMP/abba_out.txt, and only the bench step
  ever wrote it. Record the reason and the compiler output there.
- Reword the gate's error. It establishes "cc or rustc could not compile and
  run a trivial program"; bad RAM is named as one possible cause rather than
  asserted as the diagnosis.

Comments, each previously at odds with the code or with each other:

- the gate blamed bad RAM while the CARGO_BUILD_JOBS comment blamed memory
  pressure for the same symptom. The latter now describes OOM as it actually
  presents (SIGKILL, or an allocation failure) and names jemalloc-sys's
  CARGO_MAKEFLAGS forwarding, which is what makes the cap bind its nested make.
- drop the unmeasured "~10 min dual build", and annotate the 3 min 56 s ETA
  reference as a pre-cap measurement that CARGO_BUILD_JOBS=8 will raise.
- the no-offer error and the env header now list reliability, gpu_frac and
  cuda_max_good, which they had drifted from.
- state the gate's scope: it does not exercise /opt/lambda-vm-sysroot, and a
  1 s compile surfaces marginal RAM only sometimes.

* fix(bench-gpu): tell the operator to wait before re-rolling the box

Both host-fault messages said "Re-run /bench-gpu to reroll the box", but
offer selection is deterministic — `sort_by(.dph_total) | reverse | .[0]`
with no machine_id exclusion — so an immediate re-run can re-pick the same
machine once it relists and fail identically. Say to wait a few minutes
instead, and say why, so the advice matches what the picker actually does.

The ssh-255 message is left as an immediate retry: a transport failure is
not a verdict on the host, so there is nothing to roll off.

Still not an automated reroll (the sibling gpu-tests.yml carries a TRIED
machine_id list for that); this only stops the message promising something
the selection logic does not do.

---------

Co-authored-by: Mauro Toscano <12560266+MauroToscano@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant