model -Zthreads in the rustc cache adapter

[pimeys]

Every compilation in a workspace whose cargo config sets -Z threads=N (parallel rustc frontend) bypasses the cache with unknown-flag: rustc flag is not modeled by the cache adapter: -Zthreads=16.

Repro: add rustflags = "-Z threads=16" to .cargo/config.toml on a nightly toolchain and run mbx check with MBX_BYPASS_LOG set. All compilations bypass.

-Zthreads only affects codegen scheduling, so it looks safely keyable like other modeled -Z flags. This flag is increasingly common in large workspaces on many-core machines, so modeling it would unlock caching for exactly the builds that benefit most.

Observed on mbx 1.1.0 (x86_64-unknown-linux-musl), rustc 1.97.0-nightly.

[jdx]

Thanks for the report — confirmed. The rustc cache adapter was conservatively rejecting every -Z option except shell-argfiles, so -Zthreads=N caused the whole workspace to bypass.

I opened #238 to model it as keyed action input. The fix canonicalizes both -Zthreads=N and -Z threads=N, keeps different thread counts in distinct cache keys, and also covers rustc’s newer --jobs-frontend=N spelling. The full CI matrix is green.

[pimeys]

Follow-up with numbers from a large workspace (~1200 compilations, 96-core shared box):

#238 works: -Z threads=16 no longer bypasses. A cold workspace check through main@922e9ede already matches plain cargo (134s vs 157s), and it stores 1.3 GiB for later runs.

mold remains: -C link-arg=-fuse-ld=mold causes 164 unmodeled-link-argument bypasses per check: every compilation that links (build-script binaries, proc macros, bin/test targets).

Local patch (carried in our nix package until something lands here, ~120 lines across 7 files): the adapter parses -fuse-ld=<name> into a structured argument instead of opaque text, admits it for native links only (wasm links, test-only outputs, and conflicting duplicate selections still bypass), and keys it as text. On the runtime side the linker identity probe resolves ld.<name> (absolute paths used directly) and pins path: version-line in linker_version, so on content-addressed toolchains the store path alone pins the exact linker build. The selection also joins the identity memoization key, which required allowlisting it in the cache agent's executable-identity environment check; that validation doing its job was the one non-obvious integration point.

Results with the patch, repo rustflags untouched: warm check --workspace 131s to 32s (1294 hits, 0 misses), warm build of the largest binary 311s to 40s, and the cache-restored 700 MB mold-linked binary verified running. Existing cache keys are untouched, so the change is purely additive.

If the approach looks right I can open a PR with the patch as-is.

[jdx]

Thanks for the detailed follow-up. The mold bypass is real, and this approach fits the existing safety model: -fuse-ld=<name> can be parsed as a structured linker selection, while every other opaque link argument remains conservative. Please open the PR.

One correctness detail I want to verify in review: the executable-identity memoization key should contain the resolved linker path, not only the raw selector such as mold. Resolution may depend on the compiler driver search path, COMPILER_PATH, or PATH; keying only cc + mold could reuse an identity for a different ld.mold. Resolving through the driver first, then including that absolute path in both the linker identity and its memoization key, avoids that problem.

The rest of the proposed boundaries look right: native links only, wasm and unsupported outputs still bypass, conflicting selections bypass, linkless check compilations remain cacheable, and invocations without -fuse-ld retain their existing keys. Tests should cover missing linkers, selector-dependent action keys and memoization, duplicate selections, wasm rejection, and the unchanged default-linker key.

[pimeys]

PR opened: #254

The memoization key now carries the resolved absolute linker path rather than the raw selector. Resolution asks the driver first via -print-prog-name=ld.<name>, which covers COMPILER_PATH and toolchain-internal linkers, and falls back to PATH; an absolute -fuse-ld=/path/to/ld selection is used directly. The identity's linker_version pins that same resolved path plus its version line, so the identity and its memoization key cannot disagree about which linker a link used.

Coverage in the PR: selector-dependent action keys, conflicting duplicate selections, wasm rejection, inert linkless compilations (unit); resolution failures and absolute-path selections (unit); an end-to-end restore of a mold-linked test binary through two distinct target directories (bats, skips where no ld.mold/ld.lld exists). Local gate: fmt --check, clippy --workspace --all-features --all-targets -- -D warnings, and cargo test --workspace (723 tests) all green.