Skip to content

build: parallelize native compile with xargs -P, not batch-and-wait#78

Merged
dfa1 merged 1 commit into
mainfrom
build-xargs-parallel-compile
Jul 18, 2026
Merged

build: parallelize native compile with xargs -P, not batch-and-wait#78
dfa1 merged 1 commit into
mainfrom
build-xargs-parallel-compile

Conversation

@dfa1

@dfa1 dfa1 commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Problem

The compile loop backgrounded up to $JOBS zig cc invocations, then hit a
bare wait every $JOBS iterations before starting the next batch:

i=0
for src in $SRCS; do
    zig cc -target "$ZIG_TARGET" $CFLAGS -c "$src" -o "$out" &
    i=$((i + 1))
    [ $((i % JOBS)) -eq 0 ] && wait
done
wait

Two problems with this:

  1. Batch barrier, not a work queue. Every JOBS compiles, every core
    idles until the slowest TU in that batch finishes before the next batch
    starts — no work-stealing across the boundary.
  2. Failures are swallowed. A backgrounded zig cc failing under & is
    invisible to set -e — a bare wait returns success regardless of the
    child's exit status. A broken compile silently produces no .o and only
    surfaces later as a cryptic link error (missing symbols) or, worse, a
    link that "succeeds" against a stale .o left over from a previous run.

Fix

Replace the loop with xargs -P "$JOBS", which keeps all $JOBS slots
continuously saturated (no batch boundary) and returns non-zero if any
invocation fails, which set -e now catches immediately.

compile_one() {
    zig cc -target "$ZIG_TARGET" $CFLAGS -c "$1" -o "$WORK/$(basename "$1").o"
}
export -f compile_one
export ZIG_TARGET CFLAGS WORK

printf '%s\n' $SRCS | xargs -P "$JOBS" -I{} bash -c 'compile_one "$@"' _ {}

-I{} + bash -c '... "$@"' _ {} is the standard idiom for passing an
xargs-substituted value into bash -c as $1 rather than splicing it
unquoted into the command string.

Testing

Verified locally with zig cc (osx-aarch64 target):

  • Clean build still produces a working libzstd.dylib.
  • Injected a syntax error into xxhash.c and reran — previously this kind
    of failure was invisible; now the script exits 1 immediately with the
    compiler's error printed, before ever reaching the link step.

Addresses point 1 of the review feedback on the native build script.

The previous loop backgrounded JOBS compiles then `wait`ed for the whole
batch before starting the next JOBS - one slow TU idles every other core
until the batch drains. Worse, a failing background zig cc under `&` was
invisible to set -e (bare `wait` returns 0 regardless of child exit
status), so a broken compile silently produced no .o and only surfaced
later as a cryptic link error.

xargs -P keeps all JOBS slots continuously saturated (no batch boundary)
and propagates a non-zero exit from any invocation, which set -e now
catches immediately.
@dfa1
dfa1 merged commit 955dc96 into main Jul 18, 2026
1 check passed
@dfa1
dfa1 deleted the build-xargs-parallel-compile branch July 18, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant