Skip to content

chore: stop xtask from bloating target/ and slowing the gate#306

Merged
mcritzjam merged 1 commit into
mainfrom
mmansour/speed-up-gate
May 19, 2026
Merged

chore: stop xtask from bloating target/ and slowing the gate#306
mcritzjam merged 1 commit into
mainfrom
mmansour/speed-up-gate

Conversation

@mohamedmansour
Copy link
Copy Markdown
Contributor

The local quality gate (cargo xtask check) was taking 18+ minutes on long-lived working copies, even though the actual compile + test work was only ~2 minutes. The root cause was the workspace target/ directory growing to 160+ GB / 1.1M+ files from accumulated incremental compilation state and per-feature fingerprint variants. Cargo then spent most of every subsequent invocation just stat'ing fingerprint files.

Two changes:

  • Force CARGO_INCREMENTAL=0 on every cargo invocation that xtask spawns. xtask never runs an inner edit-compile loop — every cargo call it makes (gate steps, examples build, bench, E2E, packaging) is one-shot, so incremental compilation just produces garbage that cargo then has to scan on the next run. CI was already immune to this via Swatinem/rust-cache@v2 (which sets the same env). Real inner-loop development with cargo build / cargo test outside xtask is unaffected.

  • Switch the gate's bench-validate step to --profile=dev. The step exists only to run criterion's --test smoke path, but the default bench profile inherits the release profile's lto = true, codegen-units = 1, so it was recompiling the entire microsoft-webui graph with full LTO just to assert criterion's entry point loads. Dev profile reuses unit-test artifacts and drops that step from ~44s to ~2s. Real benchmark measurements via cargo xtask bench still use the unchanged bench profile, so reported numbers remain accurate.

Measured on a working copy where target/ had grown organically:

before (bloated target/, 160 GB):  18m23s
after  (bloated target/, 160 GB):  N/A — change forces clean rebuild
after  (clean target/):             42s   (first run)
after  (clean target/):             35s   (subsequent runs, no growth)

bench-validate step alone:

before:  ~44s  (bench profile, LTO + codegen-units=1)
after:   ~2-3s (dev profile, reuses test artifacts)

The local quality gate (cargo xtask check) was taking 18+ minutes on
long-lived working copies, even though the actual compile + test work
was only ~2 minutes. The root cause was the workspace target/ directory
growing to 160+ GB / 1.1M+ files from accumulated incremental
compilation state and per-feature fingerprint variants. Cargo then
spent most of every subsequent invocation just stat'ing fingerprint
files.

Two changes:

* Force `CARGO_INCREMENTAL=0` on every cargo invocation that xtask
  spawns. xtask never runs an inner edit-compile loop — every cargo
  call it makes (gate steps, examples build, bench, E2E, packaging) is
  one-shot, so incremental compilation just produces garbage that
  cargo then has to scan on the next run. CI was already immune to
  this via Swatinem/rust-cache@v2 (which sets the same env). Real
  inner-loop development with `cargo build` / `cargo test` outside
  xtask is unaffected.

* Switch the gate's bench-validate step to `--profile=dev`. The step
  exists only to run criterion's `--test` smoke path, but the default
  bench profile inherits the release profile's `lto = true,
  codegen-units = 1`, so it was recompiling the entire `microsoft-webui`
  graph with full LTO just to assert criterion's entry point loads.
  Dev profile reuses unit-test artifacts and drops that step from ~44s
  to ~2s. Real benchmark measurements via `cargo xtask bench` still
  use the unchanged bench profile, so reported numbers remain accurate.

Measured on a working copy where target/ had grown organically:

```
before (bloated target/, 160 GB):  18m23s
after  (bloated target/, 160 GB):  N/A — change forces clean rebuild
after  (clean target/):             42s   (first run)
after  (clean target/):             35s   (subsequent runs, no growth)
```

bench-validate step alone:

```
before:  ~44s  (bench profile, LTO + codegen-units=1)
after:   ~2-3s (dev profile, reuses test artifacts)
```

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mohamedmansour mohamedmansour requested a review from akroshg May 19, 2026 05:20
Copy link
Copy Markdown
Contributor

@mcritzjam mcritzjam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dev prod ftw

@mcritzjam mcritzjam merged commit 9fc32fb into main May 19, 2026
21 checks passed
@mcritzjam mcritzjam deleted the mmansour/speed-up-gate branch May 19, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants