build: parallelize native compile with xargs -P, not batch-and-wait#78
Merged
Conversation
The previous loop backgrounded JOBS compiles then `wait`ed for the whole batch before starting the next JOBS - one slow TU idles every other core until the batch drains. Worse, a failing background zig cc under `&` was invisible to set -e (bare `wait` returns 0 regardless of child exit status), so a broken compile silently produced no .o and only surfaced later as a cryptic link error. xargs -P keeps all JOBS slots continuously saturated (no batch boundary) and propagates a non-zero exit from any invocation, which set -e now catches immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The compile loop backgrounded up to
$JOBSzig ccinvocations, then hit abare
waitevery$JOBSiterations before starting the next batch:Two problems with this:
JOBScompiles, every coreidles until the slowest TU in that batch finishes before the next batch
starts — no work-stealing across the boundary.
zig ccfailing under&isinvisible to
set -e— a barewaitreturns success regardless of thechild's exit status. A broken compile silently produces no
.oand onlysurfaces later as a cryptic link error (missing symbols) or, worse, a
link that "succeeds" against a stale
.oleft over from a previous run.Fix
Replace the loop with
xargs -P "$JOBS", which keeps all$JOBSslotscontinuously saturated (no batch boundary) and returns non-zero if any
invocation fails, which
set -enow catches immediately.-I{}+bash -c '... "$@"' _ {}is the standard idiom for passing anxargs-substituted value into
bash -cas$1rather than splicing itunquoted into the command string.
Testing
Verified locally with
zig cc(osx-aarch64 target):libzstd.dylib.xxhash.cand reran — previously this kindof failure was invisible; now the script exits
1immediately with thecompiler's error printed, before ever reaching the link step.
Addresses point 1 of the review feedback on the native build script.