Skip to content

Coverage Nightly races on the shared ~/.cargo/bin (cargo-llvm-cov 'No such file or directory') — security job already has per-job CARGO_HOME, coverage does not #2353

Description

@noahgift

Coverage Nightly failed 2026-07-31 and reported an empty coverage figure into #2307:

**Line coverage: ** (target ≥ 95%)

That is a missing measurement, not a low one — the same class as project_coverage_floor_measurement_broken. It failed closed, which is correct behaviour.

Root cause

15:31:01  Compiling cargo-llvm-cov v0.6.24
15:36:30  error: could not compile `aprender-train-wasm` (lib test)
          could not execute process `/home/noah/.cargo/bin/cargo-llvm-cov …` (never executed)
          No such file or directory (os error 2)
15:38:51  make: *** [Makefile:311: coverage] Error 1

cargo llvm-cov invokes /home/noah/.cargo/bin/cargo-llvm-cov as a rustc wrapper. mac-server hosts 16 runners sharing one $HOME/.cargo/bin, so a concurrent cargo install cargo-llvm-cov replaces that binary out from under an in-flight build. Classic shared-CARGO_HOME race.

The fix already exists — it just isn't applied here

The security job runs with per-job isolation:

CARGO_HOME: /tmp/cargo-home-security-intel-clean-room-14

The coverage job does not, and uses the shared $HOME/.cargo. Giving coverage (and any other job that installs a cargo subcommand) the same treatment removes the race.

Trigger

The failure landed in a 15:20–15:40 window immediately after 16 previously-stopped runners were restarted, which flooded the box (load average peaked ~27 on 32 threads). So this is load-triggered, not load-caused — the race is always present and simply becomes likely under concurrency. It will recur on any busy night.

The same window also produced a contention breach in the GaussianNB speed beat (ratio 0.535 > 0.50, both fit and predict phases up together — the harness's own contention discriminator). That one is expected behaviour and is documented in the beat; this one is not.

Suggested work

  1. Give the coverage job a per-job CARGO_HOME, matching security.
  2. Audit every job that runs cargo install for the same exposure.
  3. Consider pre-baking cargo-llvm-cov into the sovereign-ci image so no job installs it at runtime (see feedback_check_baked_container_env_before_ci_changes — the image already bakes RUSTFLAGS/sccache/nextest).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions