perf(pnpm 11): consider streaming tarball extraction during download

[zkochan]

Tracking the TypeScript counterpart of #14021 (pacquet extracts pinned gzip tarballs ≥ 4 MiB while their bodies download, so the biggest packages — whose downloads finish last on a shared link — no longer serialize their whole verify+extract after the install's final byte; measured −94 ms mean on the official benchmark's trusted-lockfile row).

The TS fetcher buffers the full response and then runs gunzipSync + parseTarball on the complete buffer. A streaming counterpart would thread the response stream through an incremental ssri checker and a streaming gunzip/tar parser, with the same guards #14021 grew under review: admission bounded to CPU-quota-aware parallelism, first-attempt-only streaming so retry diagnostics stay on the buffered path, integrity verdict evaluated over the complete body and outranking parse errors.

Whether the win carries over depends on how much of pnpm 11's extract cost sits after the last byte — worth measuring before building, same method as pnpm/pnpm#14015.

Analogous to #14017 (pnpm 11 adoption of the lockfile memo).

🤖 Generated with Claude Code

[github-actions[bot]]

Oz triage is running now and will classify this issue into one of: Ready to implement, Ready to spec, Needs info, or Wait to implement, and apply the matching pnpm state: label.

Follow along in Oz: https://oz.warp.dev/runs/01a01c09-1295-7870-a38f-f86a135960d0

[github-actions[bot]]

Triage result

• Issue: #14023 perf(pnpm 11): consider streaming tarball extraction during download
• State: Ready to spec
• Applied label: state: needs design
• Product / parity: TypeScript CLI (pnpm11/store/cafs), tracking the pacquet counterpart in pnpm/pnpm#14021. This is inherently cross-stack parity work — a design for the TS side should account for keeping behavior aligned with whatever pacquet ultimately ships.
• Rationale: The issue itself flags that the core premise ("does the win carry over to pnpm 11?") is unmeasured and explicitly says it's "worth measuring before building." Neither pnpm/pnpm#14021 (the Rust analog) nor this TS counterpart has a merged implementation yet — pacquet's current download.rs/extract.rs still buffer the full response body before any extraction begins; only the post-buffer decode/tar-walk streams to avoid materializing the decompressed archive, not extraction concurrent with the download itself. On the TS side, pnpm11/store/cafs/src/addFilesFromTarball.ts and parseTarball.ts confirm the description: the whole tarball is buffered, then gunzipSync'd in one call, then walked by a hand-rolled binary tar parser that does random-access offset reads into the fully-materialized buffer — that parser isn't structured for incremental/streaming consumption today, so supporting true stream-while-downloading would need real design work (streaming gunzip + incremental ssri verification + an extractor that no longer assumes random access, plus the admission-control/retry/integrity-ordering guards the issue calls out from perf(tarball): extract big tarballs while they download #14021's review). This mirrors the readiness of its named analogs, #14015 and #14017, both already labeled state: needs design for the same reasons (clear goal, cohesive with the product, but material technical decisions and feasibility measurement remain before implementation).
• Next step: Measure how much of pnpm 11's extract cost lands after the tarball's last byte (same method as #14015) to confirm the win exists on this stack, then spec the streaming redesign of parseTarball/addFilesFromTarball in tandem with whatever design pacquet settles on in #14021.