chore: respect Cargo parallelism settings for native release builds - #5344
Conversation
| # build native libs for amd64 architecture Linux/MacOS on a Linux/amd64 machine/container | ||
| core-amd64-libs: | ||
| cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build -j 2 --release $(FEATURES_ARG) | ||
| cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build --release $(FEATURES_ARG) |
There was a problem hiding this comment.
[P2] Could we forward CARGO_BUILD_JOBS into both release-builder containers before removing this cap? The documented release workflow invokes dev/release/build-release-comet.sh, whose two docker run commands impose --memory 24g and --cpus 6 but do not pass CARGO_BUILD_JOBS or mount the host Cargo configuration. Their entrypoint then runs these Makefile targets, so setting CARGO_BUILD_JOBS=2 on the host does not reach Cargo inside either container. As a result, the actual release build loses its existing two-job limit and cannot use the override described in this PR, while the release profile enables debug information and ThinLTO. The green native CI jobs call cargo build --profile ci directly, so they do not exercise this Dockerized release path. Please propagate the variable through both docker run invocations, ideally with an overridable default of 2.
There was a problem hiding this comment.
addressed via --env "CARGO_BUILD_JOBS=${CARGO_BUILD_JOBS:-2}"
|
Note that I think this was constrained because we’d occasionally see OOMs in the container with too much parallelism, but we’ll see how it goes for the next release. |
Which issue does this PR close?
No issue filed.
Rationale for this change
The
core-amd64-libsandcore-arm64-libstargets hardcodecargo build -j 2, limiting compilation to two concurrent jobs regardless of available CPU resources.This also overrides Cargo's standard
CARGO_BUILD_JOBSenvironment variable andbuild.jobsconfiguration, preventing users from controlling parallelism through existing Cargo mechanisms.Removing the hardcoded limit lets Cargo automatically determine the appropriate level of parallelism while preserving the ability to configure it explicitly:
What changes are included in this PR?
Remove
-j 2from all four Cargo invocations incore-amd64-libsandcore-arm64-libs, including optional macOS cross-compilation builds.How are these changes tested?
Verified the generated commands for both architectures:
Also verified that
CARGO_BUILD_JOBS=7is passed through to all four Cargo invocations using mockedcargoandrustupbinaries.