Coverage Nightly failed 2026-07-31 and reported an empty coverage figure into #2307:
**Line coverage: ** (target ≥ 95%)
That is a missing measurement, not a low one — the same class as project_coverage_floor_measurement_broken. It failed closed, which is correct behaviour.
Root cause
15:31:01 Compiling cargo-llvm-cov v0.6.24
15:36:30 error: could not compile `aprender-train-wasm` (lib test)
could not execute process `/home/noah/.cargo/bin/cargo-llvm-cov …` (never executed)
No such file or directory (os error 2)
15:38:51 make: *** [Makefile:311: coverage] Error 1
cargo llvm-cov invokes /home/noah/.cargo/bin/cargo-llvm-cov as a rustc wrapper. mac-server hosts 16 runners sharing one $HOME/.cargo/bin, so a concurrent cargo install cargo-llvm-cov replaces that binary out from under an in-flight build. Classic shared-CARGO_HOME race.
The fix already exists — it just isn't applied here
The security job runs with per-job isolation:
CARGO_HOME: /tmp/cargo-home-security-intel-clean-room-14
The coverage job does not, and uses the shared $HOME/.cargo. Giving coverage (and any other job that installs a cargo subcommand) the same treatment removes the race.
Trigger
The failure landed in a 15:20–15:40 window immediately after 16 previously-stopped runners were restarted, which flooded the box (load average peaked ~27 on 32 threads). So this is load-triggered, not load-caused — the race is always present and simply becomes likely under concurrency. It will recur on any busy night.
The same window also produced a contention breach in the GaussianNB speed beat (ratio 0.535 > 0.50, both fit and predict phases up together — the harness's own contention discriminator). That one is expected behaviour and is documented in the beat; this one is not.
Suggested work
- Give the coverage job a per-job
CARGO_HOME, matching security.
- Audit every job that runs
cargo install for the same exposure.
- Consider pre-baking
cargo-llvm-cov into the sovereign-ci image so no job installs it at runtime (see feedback_check_baked_container_env_before_ci_changes — the image already bakes RUSTFLAGS/sccache/nextest).
Coverage Nightly failed 2026-07-31 and reported an empty coverage figure into #2307:
That is a missing measurement, not a low one — the same class as
project_coverage_floor_measurement_broken. It failed closed, which is correct behaviour.Root cause
cargo llvm-covinvokes/home/noah/.cargo/bin/cargo-llvm-covas a rustc wrapper. mac-server hosts 16 runners sharing one$HOME/.cargo/bin, so a concurrentcargo install cargo-llvm-covreplaces that binary out from under an in-flight build. Classic shared-CARGO_HOME race.The fix already exists — it just isn't applied here
The
securityjob runs with per-job isolation:The coverage job does not, and uses the shared
$HOME/.cargo. Giving coverage (and any other job that installs a cargo subcommand) the same treatment removes the race.Trigger
The failure landed in a 15:20–15:40 window immediately after 16 previously-stopped runners were restarted, which flooded the box (load average peaked ~27 on 32 threads). So this is load-triggered, not load-caused — the race is always present and simply becomes likely under concurrency. It will recur on any busy night.
The same window also produced a contention breach in the GaussianNB speed beat (
ratio 0.535 > 0.50, both fit and predict phases up together — the harness's own contention discriminator). That one is expected behaviour and is documented in the beat; this one is not.Suggested work
CARGO_HOME, matchingsecurity.cargo installfor the same exposure.cargo-llvm-covinto thesovereign-ciimage so no job installs it at runtime (seefeedback_check_baked_container_env_before_ci_changes— the image already bakes RUSTFLAGS/sccache/nextest).