ci: bound Windows build parallelism (intermittent OOM/SIGTERM) - #9
Merged
Conversation
The Build step set JOBS="-j" as its default and only overrode it with an explicit cap for Linux (-j 2) and macOS (-j 3), leaving the windows-2022 matrix leg with a bare, unbounded -j. A bare -j/--parallel means make/cmake spawn one compiler or linker per ready translation unit; with JUCE's large TUs, plus LTO staying enabled specifically on Windows in the Configure step, that exhausts the runner's RAM and the job gets killed with SIGTERM (exit 143, "Terminated", no compiler error above it). It's intermittent - the same commit can pass one run and fail the next depending on scheduling. Pin Windows to -j 2 (matching Linux's conservative cap, since Windows keeps LTO on and each link is heavier) and change the fallback default from bare -j to -j 2 so any future unlisted matrix OS is safe too.
Collaborator
Author
Validated
Run 30156870758 — all three legs green:
Suggestive side note, not proof: the whole run took 11m48s, against 21m21s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while auditing the fleet after Caja's CI died to this exact pattern.
JOBSdefaulted to a bare-jand was overridden with an explicit cap only forLinux (
-j 2) and macOS (-j 3). Thewindows-2022leg fell through to theunbounded default.
A bare
-jmeans make/cmake spawn one compiler per ready translation unit. WithJUCE's large TUs that exhausts the runner and the job is killed with SIGTERM —
exit 143,
Terminated, and crucially no compiler error above it, so it doesnot read as a build failure. It is intermittent by nature: the same commit can
pass one run and fail the next.
Windows is the leg where it matters most, because the Configure step
deliberately keeps LTO on for Windows only (
if [ "$RUNNER_OS" != "Windows" ]; then EXTRA="-DBOMBO_LTO=OFF"; fi), which makes each link heavier than theLTO-off Linux/macOS builds.
Now every OS is explicitly bounded and the fallback default is
-j 2rather thanunbounded, so a future runner added to the matrix is safe by default.
CI-only. No product code touched, so this does not intrude on the scope freeze.
Background: this cost about a day on Caja before it was traced — the tell is that
the same commit passed on its
pull_requestrun and failed on itspushrun.🤖 Generated with Claude Code