Skip to content

chore: respect Cargo parallelism settings for native release builds - #5344

Merged
sunchao merged 2 commits into
apache:mainfrom
pingzh:perf/respect-cargo-native-build-parallelism
Aug 14, 2026
Merged

chore: respect Cargo parallelism settings for native release builds#5344
sunchao merged 2 commits into
apache:mainfrom
pingzh:perf/respect-cargo-native-build-parallelism

Conversation

@pingzh

@pingzh pingzh commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No issue filed.

Rationale for this change

The core-amd64-libs and core-arm64-libs targets hardcode cargo build -j 2, limiting compilation to two concurrent jobs regardless of available CPU resources.

This also overrides Cargo's standard CARGO_BUILD_JOBS environment variable and build.jobs configuration, preventing users from controlling parallelism through existing Cargo mechanisms.

Removing the hardcoded limit lets Cargo automatically determine the appropriate level of parallelism while preserving the ability to configure it explicitly:

CARGO_BUILD_JOBS=2 make core-amd64-libs

What changes are included in this PR?

Remove -j 2 from all four Cargo invocations in core-amd64-libs and core-arm64-libs, including optional macOS cross-compilation builds.

How are these changes tested?

Verified the generated commands for both architectures:

make -n core-amd64-libs core-arm64-libs
make -n HAS_OSXCROSS=1 core-amd64-libs core-arm64-libs

Also verified that CARGO_BUILD_JOBS=7 is passed through to all four Cargo invocations using mocked cargo and rustup binaries.

Comment thread Makefile
# build native libs for amd64 architecture Linux/MacOS on a Linux/amd64 machine/container
core-amd64-libs:
cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build -j 2 --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build --release $(FEATURES_ARG)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Could we forward CARGO_BUILD_JOBS into both release-builder containers before removing this cap? The documented release workflow invokes dev/release/build-release-comet.sh, whose two docker run commands impose --memory 24g and --cpus 6 but do not pass CARGO_BUILD_JOBS or mount the host Cargo configuration. Their entrypoint then runs these Makefile targets, so setting CARGO_BUILD_JOBS=2 on the host does not reach Cargo inside either container. As a result, the actual release build loses its existing two-job limit and cannot use the override described in this PR, while the release profile enables debug information and ThinLTO. The green native CI jobs call cargo build --profile ci directly, so they do not exercise this Dockerized release path. Please propagate the variable through both docker run invocations, ideally with an overridable default of 2.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed via --env "CARGO_BUILD_JOBS=${CARGO_BUILD_JOBS:-2}"

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sunchao
sunchao merged commit fbdc267 into apache:main Aug 14, 2026
51 checks passed
@mbutrovich

Copy link
Copy Markdown
Contributor

Note that I think this was constrained because we’d occasionally see OOMs in the container with too much parallelism, but we’ll see how it goes for the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants