docs(pm): benchmark reference for cold and warm installs

[elrrrrrrr]

Purpose

Keep a stable, linkable reference for utoo package-manager benchmark results and methodology. Future PRs should link this issue instead of quoting results from the legacy pm-bench-all harness.

Benchmark contract

Compare these three user-visible states:

1. Full cold (p0): no lockfile, no manifest cache, no tarball/package store, no node_modules.
2. Lock-only cold (p3): lockfile exists, but tarball/package store and node_modules are removed.
3. Warm (p4): lockfile and tarball/package store are populated; only node_modules is removed.

Do not use “no lockfile but warm manifest cache”. Bun keeps manifests fresh for five minutes without revalidation, so that state disproportionately benefits Bun and is not one of the intended scenarios.

The phase harness uses explicit per-PM stores, restores each PM's own lockfile, disables lifecycle scripts, alternates PM order between rounds, and records network traffic so an accidentally cold “warm” cell can be rejected.

GitHub Actions · npmjs

• Harness branch/commit: codex/aube-phase-benchmark / 7a76aa3
• Project: ant-design
• Registry: https://registry.npmjs.org
• Workflow: https://github.com/utooland/utoo/actions/runs/29797404521
• Sampling: 3 interleaved rounds for p0/p3; 5 for p4
• aube: 1.29.1 with trustPolicy: off

Linux (ubuntu-latest)

Mode	utoo	Bun	aube
Full cold (p0)	8.13s ± 0.33s	9.70s ± 0.24s	17.54s ± 0.74s
Lock-only cold (p3)	7.22s ± 1.24s	7.52s ± 0.04s	8.37s ± 1.04s
Warm (p4)	2.25s ± 0.12s	3.58s ± 0.05s	2.59s ± 0.13s

macOS (macos-latest, Apple Silicon)

Mode	utoo	Bun	aube
Full cold (p0)	19.19s ± 2.43s	22.81s ± 1.29s	55.28s ± 5.78s
Lock-only cold (p3)	12.03s ± 0.53s	15.58s ± 3.94s	31.31s ± 3.12s
Warm (p4)	3.81s ± 0.19s	5.37s ± 0.22s	14.76s ± 0.26s

The pnpm cells from this workflow are intentionally omitted: ant-design's .npmrc contains package-lock=false, which pnpm 10 mapped to lockfile=false, so those cells had no pnpm-lock.yaml and were not valid p3/p4 measurements. Harness commit f5210b24 now forces --config.lockfile=true.

Corrected pnpm validation

• Hardened harness commit: 1e88fa2b
• Workflow: https://github.com/utooland/utoo/actions/runs/29809892297

Platform	Full cold (p0)	Lock-only cold (p3)	Warm (p4)
Linux	26.20s ± 0.81s	18.01s ± 1.03s	7.19s ± 0.02s
macOS	70.58s ± 13.74s	57.25s ± 15.07s	31.88s ± 2.58s

The Linux warm cell retained a ~1 MB pnpm-lock.yaml and ~1.45 GB store while transferring only ~7 KB RX / 18 KB TX. This confirms the corrected p4 cell is genuinely warm. The same run rejected aube p3/p4 because the expected seeded lockfile was absent rather than timing stale state.

poolab Linux · npmmirror direct connection

• Date: 2026-07-21
• Host: Linux x86_64 (poolab)
• Registry: https://registry.npmmirror.com
• Network: direct connection; no HTTP_PROXY, HTTPS_PROXY, NO_PROXY, npm proxy, macmini host entry, or macmini shell configuration was present
• Versions: utoo 1.1.3, Bun 1.3.14, pnpm 10.34.5, aube 1.29.1
• Sampling: 3 interleaved rounds for p0/p3; 5 for p4

Mode	utoo	Bun	pnpm	aube
Full cold (p0)	7.84s ± 2.17s	13.21s ± 2.36s	39.32s ± 16.56s	27.63s ± 6.47s
Lock-only cold (p3)	6.64s ± 1.64s	8.62s ± 3.57s	15.14s ± 0.77s	6.42s ± 0.31s
Warm (p4)	1.09s ± 0.05s	2.72s ± 0.05s	7.89s ± 0.38s	1.60s ± 0.07s

For the corrected pnpm warm run, network RX was ~4 KB and the lockfile/store were present, confirming it is a true warm measurement. The p3 utoo/aube cell was repeated for 9 paired, order-alternating rounds after hardening the harness to reject missing locks and failed prepares. Its paired utoo-minus-aube delta was +0.214s with SE 0.506s (t=0.42), so the small mean difference is noise: treat utoo and aube as tied for p3. Utoo is clearly fastest for p0 and p4.

This poolab result is a separate network/topology view and should not be mixed statistically with GitHub-hosted runner results.

Historical confirmation

PR #3199 showed the same utoo-vs-Bun direction under the phase-isolated methodology: Linux p0 7.71s vs 9.51s, Linux p4 2.40s vs 3.68s, and macOS p4 3.76s vs 4.41s.

Reference: #3199

Why the legacy pm-bench-all result is invalid

Run https://github.com/utooland/utoo/actions/runs/29740213943 used the older bench/pm-bench.sh harness. Its “warm” preparation runs git clean -dfx, deleting the generated lockfile while retaining global caches. It therefore measures fresh resolution with a warm cache and exposes Bun's five-minute manifest fast path. It also uses shared/default cache locations and does not pair/interleave rounds as strictly as pm-bench-phases.sh.

Do not cite that run as a utoo-vs-Bun comparison.

aube failure diagnosis

The legacy run's aube 1.29.1 cells failed with ERR_AUBE_TRUST_DOWNGRADE: aube's default no-downgrade supply-chain policy rejected @swc/core@1.9.2. This is a policy rejection, not a timing failure. The phase harness synthesizes:

trustPolicy: off

That permits all PMs to install the same dependency graph and produced the valid aube results above.

Follow-ups

• Retire or clearly deprecate pm-bench-all for performance claims; use pm-bench-phases for PR evidence.
• Add aube with explicit XDG cache/store isolation, lock restoration, and trustPolicy: off.
• Add pnpm with an explicit store and force lockfile generation even when an npm project sets package-lock=false.
• Capture failed command output and skip invalid JSON cells instead of crashing the summary.
• Reject missing seeded lockfiles and abort on prepare failures so stale stores cannot masquerade as cold installs (1e88fa2b).
• Add a Linux-only manual-dispatch option so registry benchmarks do not need to consume a macOS runner.