Opt-in deflate compression for file-backed blobs (harper#2443)

[kriszyp]

Opt-in deflate compression for file-backed blobs (harper#2443)

Adds opt-in, per-content-type deflate compression for Harper's file-backed blobs, and
converts the compressed-blob read path to a bounded streaming inflate so a large blob no
longer buffers its whole inflated content in memory. Companion to harper-pro#795, which preserves the codec across replication.

Compression is off unless configured. An operator opts in per content type through a new
storage.blobs.compression map; absent config means no blob is ever compressed. The on-disk
format is unchanged: type byte 1 (DEFLATE_TYPE) is the same one every v5.2.x reader already
inflates, and the 8-byte header still records the uncompressed size for both forms, so no
version fence is needed and every existing completeness check keeps working.

What changed

• Streaming inflate read path (prerequisite). A full streaming read of a compressed blob
  previously called blob.bytes() and buffered the entire inflated content as one chunk.
  stream() now inflates incrementally with a hard output ceiling at the declared size, so a
  large compressed blob streams with bounded heap and a header that understates its size is
  refused before it can emit past the bound.
• Opt-in config. storage.blobs.compression is a map of content type → { codec, threshold }
  or false, with exact-type-beats-type/*-beats-default precedence and a size threshold.
  Already-compressed types (application/gzip, application/zip, application/zstd, image/*,
  video/*, audio/*) ship as false so an ordinary upload of those is never re-compressed.
• Write path. The policy resolves in saveBlob, the single funnel every local write passes
  through (including an ordinary HTTP upload, which never sees creation options), so an uploaded
  body of a configured type is compressed on disk. An explicit compress option or a
  pre-compressed source still takes precedence. Unknown-size streamed writes never compress (a
  threshold cannot be evaluated without a size).
• Stored-codec transfer support. openStoredBlobBody (sender) and createBlobFromStoredBody
  (receiver) let replication move a peer's raw deflate body verbatim, each verifying the actual
  bytes by concurrent inflate so a torn or forged body can neither ship nor publish. Used by the
  harper-pro companion PR.
• Compression-aware repair/completeness. isBlobComplete, the backup capture classifier, and
  the in-place repair gates inflate to classify a compressed body, so a torn deflate body (which a
  length check cannot see, since the header records the uncompressed size) is still healed.

For the human reviewer

Framing note (a plan-review disagreement, decided by the task owner). The step-6 planning
review preferred temp-file + atomic rename for new compressed writes. The owner ruled to keep
in-place publication — it is what lets a read stream a blob while it is still being
write-streamed, the contract bytes() already relies on — and to fence off the PENDING/ERROR
marker semantics (harper-pro#481) and the reclamation/orphan sweep. This PR implements that
ruling; completeness of an in-place compressed body is established through the writer lock, not by
publication atomicity.

Look hardest at the two stored-body verifiers (createStoredDeflateVerifier and openStoredBlobBody.stream), which must settle on the inflater's error/close because a mid-stream Z_DATA_ERROR drops zlib's write callback, and at readCompressedBlob's post-wait file-identity check against an in-place repair swap.

Gemini bot review — dispositions. The strict-assertion suggestion is left: the compression and blob.test.js suites use node:assert's assert.equal throughout, so the new assertions match the file's convention. The close(fd) "requires a callback" flags are declined: callback-less fs.close does not throw on the supported Node versions (verified on 26) and is the file's established fire-and-forget close (a closeSync would block the event loop). The openStoredBlobBody.stream setup-leak concern is on an unreachable path — createInflate and .on() do not throw in practice — so it is left rather than restructure a reviewed function.

Decisions worth a second look, none of them accidental:

• Unknown-size streams never compress. The response-cache path stores bodies as unsized
  streams, so the workload arguably most worth compressing gets nothing today. Reversible as a
  follow-up, but operators will measure "compression on" against it.
• Untyped blobs match default. Cached response bodies carry no blob type, so the shipped
  image/*/application/gzip exclusions cannot fire for them; a default entry compresses them.
• storedCodec decodes onto the public blob (record.blob.storedCodec). It is a deliberate,
  namespaced transfer hint (a user codec property is proven to survive), but it is de-facto
  public surface on every compressed blob; removing it later is a visible change.
• A crash-torn compressed body reads as permanent corruption (500). Compression makes a
  crash-torn body indistinguishable by length from a complete one, so an unclean shutdown yields
  permanent-classified blobs that only the repair sweep heals — the same taxonomy torn blobs use
  today, and the repair sweep is deflate-aware.
• Ranged reads inflate-and-discard up to start. A high-offset range on a compressed blob
  costs an O(start) prefix inflate. Block-framed deflate with an offset index was explicitly out
  of scope; this is policy-reversible (exclude a type, or set a size cap) without a format change.
• A shipped false built-in beats an operator's type/* wildcard. An operator who sets
  application/*: {codec: deflate} will not get application/gzip/zip/zstd compressed, because
  the exact-key built-in exclusions win over a broader wildcard. Documented in the schema and
  tested; a reviewer could reasonably want operator entries to displace built-ins.

Verification

• unitTests/resources/blobCompression.test.js (compression suite) and blob.test.js run
  together: 142 passing. Full test:unit:resources: 1915 passing / 23 pending / 0 failing.
• The acceptance-critical coverage is red-provable against the pre-change code: the bounded-heap
  streaming test fails on the bytes()-buffering implementation, and each correctness fix has a
  test that hangs or misclassifies against base and passes with the fix — the two deflate-verifier
  hangs (receive and send), the compressed-read vs in-place-repair race, and the auto-compressed
  size-mismatch write.
• configValidator compression-schema tests pass, including a nested-typo rejection case. (One
  pre-existing domainSocket path-length env failure in that suite is unrelated — the deep
  worktree path exceeds the 107-byte Unix socket limit.)
• The paired harper-pro cluster test (blobCodecPreservation.test.mjs) passes end to end.
• Pre-existing CI note: the repo-wide Format Check (prettier --check .) is red on unitTests/resources/query-array-scoping.test.js, an unformatted file merged to main by Pin element-scoping semantics of queries over array-valued properties #2437 — not touched by this PR (this branch is behind that commit). Per the task's format-only-touched-files rule it is left alone; a rebase onto a fixed main clears it.

Refs #2443

🤖 Generated with Claude Code

_{Review-Coverage: authored=claude; ran=codex; blocked=gemini(auth); declined=cursor-grok,cursor-composer,domain; rounds=7 @ 4a5a5fb}

_{Human-Review-Need: 4 @ 4a5a5fb}

[gemini-code-assist]

Code Review

This pull request implements opt-in deflate compression for file-backed blobs, including configuration validation, streaming compression/decompression, and verification of compressed bodies during replication. The review feedback recommends replacing loose equality assertions in the new test file with strict assertions to align with the repository style guide. Additionally, it identifies bugs in resources/blob.ts where asynchronous close() is called without a callback, and points out a potential resource leak if an error occurs during stream setup before entering the main try-finally block.

[gemini-code-assist]

According to the repository style guide and general rules, tests should explicitly use strict assertions like assert.strictEqual and assert.deepStrictEqual instead of loose equality assertions (e.g., assert.equal, assert.deepEqual) to prevent type-coercion bugs from being masked. Please update the assertions throughout this file to use strict equality.

References

1. Tests use plain node:assert. Use assert.strictEqual/assert.deepStrictEqual explicitly where strict semantics are needed. ^(link)
2. Use strict assertions like assert.strictEqual and assert.deepStrictEqual instead of loose equality assertions (e.g., assert.equal, assert.deepEqual) in tests to prevent type-coercion bugs (such as string-vs-number typing issues) from being masked.

[gemini-code-assist]

The asynchronous fs.close() function requires a callback argument in Node.js. Calling it without one (e.g., close(staleFd)) is deprecated and can throw a TypeError depending on the Node.js version and environment. Since closeSync is already imported and safe to use synchronously here, please use closeSync(staleFd) instead.

closeSync(staleFd);

[gemini-code-assist]

The asynchronous fs.close() function requires a callback argument in Node.js. Calling it without one (e.g., close(fd)) is deprecated and can throw a TypeError depending on the Node.js version and environment. Since closeSync is already imported and safe to use synchronously here, please use closeSync(fd) instead.

closeSync(fd);

[gemini-code-assist]

If an error is thrown during the setup of the verifier or the registration of its event listeners (lines 3638–3663), the outer try-catch block (line 3621) has already exited, and the inner try-finally block (line 3664) has not yet been entered. This could result in source and streamFd being leaked, and the file hold not being released. Consider wrapping the entire setup and consumption of the stream in a single try-finally block to ensure robust cleanup.