Skip to content

perf(pm): reserve install clone capacity#2993

Draft
elrrrrrrr wants to merge 1 commit into
exp/pm-install-clone-batch-completionsfrom
exp/pm-install-reserve-clone-extract-slots
Draft

perf(pm): reserve install clone capacity#2993
elrrrrrrr wants to merge 1 commit into
exp/pm-install-clone-batch-completionsfrom
exp/pm-install-reserve-clone-extract-slots

Conversation

@elrrrrrrr
Copy link
Copy Markdown
Contributor

@elrrrrrrr elrrrrrrr commented May 19, 2026

Summary

This is an independent p3 scheduler experiment on top of #2989.

Extract/cache population and clone/materialize both run blocking filesystem work on the global rayon pool. On a 2-core runner the current limits can queue extract workers and clone workers at the same time, so clone/materialize may still compete with cache writes even after completion batching.

This PR keeps full extract concurrency when there is no clone work, but halves the effective extract limit while clone work is queued or active.

Hypothesis

  • p3 may improve if clone/materialize is currently delayed by extract/cache-write rayon work.
  • p4 should be neutral because warm link has no extract queue.
  • If p3 regresses, extract/cache population is the critical path and this reservation is too conservative.

Validation

  • cargo fmt
  • cargo test -p utoo-pm install_scheduler
  • cargo clippy -p utoo-pm --all-targets -- -D warnings --no-deps

Bench Result

GHA Linux npmjs run 26093744012:

phase utoo wall vCtx iCtx
p3 cold install 4.84s 59.0K 39.2K
p4 warm link 1.54s 8.5K 4.4K

The p3 wall is faster than #2989 latest npmjs reference (5.29s), but ctx is worse (53.0K/32.9K -> 59.0K/39.2K). Also, p4 should be a no-op control for this PR and moved a lot, which means this run had substantial GHA variance.

Conclusion: inconclusive. This needs a rerun or poollab confirmation before keeping; the wall signal is interesting, but p4 control noise prevents attributing it to the extract-slot reservation.

@elrrrrrrr elrrrrrrr added A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR labels May 19, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dynamic extraction limit in the install scheduler to better balance filesystem-intensive tasks. Specifically, it adds an effective_extract_limit method that halves the extraction concurrency when cloning work is pending or active, ensuring that clone progress is not delayed by cache writes. The pump_extracts function was updated to respect this new limit, and corresponding unit tests were added to verify the logic. I have no feedback to provide as there were no review comments.

@github-actions
Copy link
Copy Markdown

📊 pm-bench-phases · c3f0ff8 · linux (ubuntu-latest)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM wall ±σ user sys RSS pgMinor
bun 9.53s 0.97s 10.41s 10.15s 777M 345.0K
utoo-next 7.79s 0.11s 10.62s 12.18s 964M 135.0K
utoo-npm 8.72s 1.15s 10.81s 12.23s 969M 120.5K
utoo 7.68s 0.17s 11.37s 11.18s 893M 143.6K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 14.2K 17.8K 1.19G 6M 1.86G 1.75G 1M
utoo-next 117.4K 81.6K 1.16G 4M 1.71G 1.70G 2M
utoo-npm 116.5K 79.1K 1.16G 4M 1.71G 1.70G 2M
utoo 74.8K 56.8K 1.16G 6M 1.71G 1.70G 2M

p1_resolve

PM wall ±σ user sys RSS pgMinor
bun 1.87s 0.04s 4.17s 1.02s 485M 158.1K
utoo-next 3.11s 0.08s 5.63s 2.07s 619M 89.4K
utoo-npm 3.08s 0.06s 5.70s 2.04s 612M 85.9K
utoo 3.91s 2.13s 6.38s 1.69s 642M 119.8K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 7.2K 4.9K 202M 3M 107M - 1M
utoo-next 70.3K 89.0K 199M 2M 7M 3M 2M
utoo-npm 69.9K 89.0K 199M 2M 7M 3M 2M
utoo 15.3K 18.9K 202M 3M 7M 3M 2M

p3_cold_install

PM wall ±σ user sys RSS pgMinor
bun 6.71s 0.25s 6.39s 10.02s 638M 206.3K
utoo-next 6.47s 1.19s 5.21s 11.26s 493M 63.5K
utoo-npm 6.46s 0.38s 5.24s 11.02s 506M 68.1K
utoo 5.32s 0.09s 4.93s 9.59s 523M 54.8K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 3.3K 6.8K 1018M 3M 1.76G 1.76G 1M
utoo-next 105.7K 50.9K 989M 3M 1.70G 1.70G 2M
utoo-npm 92.4K 45.0K 988M 2M 1.70G 1.70G 2M
utoo 53.0K 39.2K 988M 2M 1.70G 1.70G 2M

p4_warm_link

PM wall ±σ user sys RSS pgMinor
bun 3.34s 0.02s 0.22s 2.35s 135M 32.6K
utoo-next 2.23s 0.21s 0.55s 3.96s 80M 18.1K
utoo-npm 2.20s 0.07s 0.55s 3.92s 79M 18.9K
utoo 2.15s 0.06s 0.36s 3.43s 50M 11.5K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 253 22 5M 21K 1.91G 1.75G 1M
utoo-next 43.4K 20.5K 2K 6K 1.70G 1.70G 3M
utoo-npm 43.4K 20.4K 2K 9K 1.70G 1.70G 3M
utoo 7.7K 4.5K 2K 8K 1.71G 1.70G 3M

npmmirror.com: no output captured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant