Skip to content

perf(sourcemap-upload): skip ZIP-level DEFLATE when wire codec compresses (62% CPU, 5.9% wire)#849

Merged
BYK merged 2 commits intomainfrom
byk/sourcemap-stored-zip
Apr 25, 2026
Merged

perf(sourcemap-upload): skip ZIP-level DEFLATE when wire codec compresses (62% CPU, 5.9% wire)#849
BYK merged 2 commits intomainfrom
byk/sourcemap-stored-zip

Conversation

@BYK
Copy link
Copy Markdown
Member

@BYK BYK commented Apr 25, 2026

Closes #847.

Summary

The artifact-bundle ZIP was DEFLATE-encoded inside, then each wire chunk was re-compressed by the upload codec. Compressing already-compressed bytes is wasted work — DEFLATE has already extracted the redundancy that zstd/gzip would have extracted on the wire.

When the server advertises a wire codec, switch the ZIP to STORED so the wire codec does the work. Keep DEFLATE when no wire codec is available (the chunk-upload.no-compression kill-switch) so the upload doesn't balloon to raw bytes.

Measurements (CLI binary upload, 14.3 MiB raw)

ZIP build Wire encode Total CPU Wire bytes
Before (DEFLATE + zstd) 349 ms 4 ms 352 ms 3,818,608
After (STORED + zstd) 48 ms 85 ms 134 ms 3,592,161
Delta −218 ms (62%) −226 KB (5.9%)

Re-measured locally on master at 3f326bed. Same shape as the numbers in #847; the magnitude is slightly lower than the issue's 75% (because Bun's deflate is faster than the Python zipfile baseline I used in the original investigation).

Compatibility

  • Server: reads the archive via zipfile.ZipFile in src/sentry/models/artifactbundle.py, which transparently handles STORED + DEFLATE entries. No protocol change.
  • Symbolicator: SourceBundle::test() checks the SYSB magic header, not entry compression. No change.
  • Pre-zstd / kill-switch servers (gzip-only or no compression): pickUploadEncoding returns gzip or undefined. STORED is only used when a wire codec is available, so plain uploads keep DEFLATE and stay small.
  • Direct callers of ZipWriter.create(): default is compression: "deflate" — no behavior change.

Tests

  • Property-based round-trip tests parameterized over both compression modes via describe.each.
  • Unit tests reading the local-file-header method byte (0=STORED, 8=DEFLATE) on both ZipWriter and buildArtifactBundle.
  • Sanity test asserting STORED is larger than DEFLATE on redundant input — would fail if ZIP layout drifts or input compressibility regresses.
  • Existing tests (27 in zip + sourcemaps test files; 56 across all sourcemap-related suites) continue to pass.

Implementation notes

  • New ZipCompression type exported from src/lib/sourcemap/zip.ts.
  • pickUploadEncoding is lifted from inside uploadArtifactBundle to the top of uploadSourcemaps so the ZIP construction can see the codec choice. The encoding is then passed through to both buildArtifactBundle (for STORED-vs-DEFLATE) and uploadArtifactBundle (for wire compression).
  • No env var escape hatch — the conditional on pickUploadEncoding(serverOptions.compression) is itself the safety net for any deployment that opts out of wire compression.

…sses

The artifact-bundle ZIP was DEFLATE-encoded at level 6, then each wire
chunk was re-compressed by the upload codec (zstd or gzip). Compressing
already-compressed bytes wastes CPU and barely helps wire size — DEFLATE
already extracts ~93% of the redundancy, so the per-chunk codec has
nothing left to do.

When the server advertises a wire codec, switch the ZIP to STORED so
the wire codec does the actual compression. Two effects on the CLI's
own bin.js + bin.js.map (~14 MiB raw):

  - CPU: 352 ms → 134 ms  (62% saved)
  - Wire: 3,818,608 B → 3,592,161 B  (5.9% saved, 226 KB)

When the server's compression list is empty (the
`chunk-upload.no-compression` kill-switch or unsupported codecs only),
keep DEFLATE so the wire payload doesn't balloon to 14 MiB raw.

Implementation:
  - Add `ZipCompression` type + `compression` option to
    `ZipWriter.create()` (default `"deflate"` keeps existing direct
    callers' behavior unchanged).
  - Lift `pickUploadEncoding` from inside `uploadArtifactBundle` to
    `uploadSourcemaps` so the ZIP construction can see the codec
    choice. Pass it through to `buildArtifactBundle`.
  - Server reads the archive via `zipfile.ZipFile`, which transparently
    handles STORED + DEFLATE entries — no protocol change.
  - Symbolicator's `SourceBundle::test()` cares about the SYSB magic
    header, not entry compression — no change there either.

Tests:
  - Property tests (round-trip via `unzip -p`) parameterized over both
    compression modes.
  - Unit tests for the local-file-header method byte (0=STORED,
    8=DEFLATE) on both `ZipWriter` and `buildArtifactBundle`.
  - Sanity test that STORED is larger than DEFLATE for redundant input
    (would catch ZIP-layout regressions).

Refs #847
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 25, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://cli.sentry.dev/_preview/pr-849/

Built to branch gh-pages at 2026-04-25 19:40 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 25, 2026

Codecov Results 📊

6007 passed | Total: 6007 | Pass Rate: 100% | Execution Time: 0ms

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +57
Passed Tests 📈 +57
Failed Tests
Skipped Tests

All tests are passing successfully.

❌ Patch coverage is 32.56%. Project has 12899 uncovered lines.
✅ Project coverage is 75.57%. Comparing base (base) to head (head).

Files with missing lines (2)
File Patch % Lines
src/lib/api/sourcemaps.ts 17.86% ⚠️ 23 Missing
src/lib/sourcemap/zip.ts 60.00% ⚠️ 6 Missing
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    74.84%    75.57%    +0.73%
==========================================
  Files          285       286        +1
  Lines        53047     52804      -243
  Branches         0         0         —
==========================================
+ Hits         39699     39905      +206
- Misses       13348     12899      -449
- Partials         0         0         —

Generated by Codecov Action

@BYK BYK merged commit aec88a3 into main Apr 25, 2026
26 checks passed
@BYK BYK deleted the byk/sourcemap-stored-zip branch April 25, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(sourcemap-upload): switch artifact-bundle ZIP to STORED to let per-chunk zstd actually compress (75% CPU saved, 5% wire saved)

1 participant