Add HttpServer response compression middleware - #6898
Conversation
Implements Accept-Encoding negotiation (RFC 9110), skip logic, Vary handling, and ETag weakening in shared http modules, with a new HttpPlatform compression primitive implemented via native APIs on Node.js, Bun, Deno, and web handlers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: cd0a61e The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
ℹ️ One rough edge worth a look, otherwise solid.
Reviewed changes
- Compression middleware in
HttpMiddleware— negotiatesAccept-Encodingper RFC 9110 (q-values for acceptability, server order for ranking, wildcard matching), skips ineligible responses (status, existing encoding,no-transform, content type,minSize, empty/FormData bodies), managesVary: Accept-Encoding, and applies via the pre-response handler mechanism. HttpPlatformcompression primitive — newCompressioninterface withalgorithmsadvertisement andcompressResponse;wrapCompressionlayers onContent-Encoding,Vary, and ETag weakening;makeCompressionhelper for Web-stream-based platforms with optional sync one-shot.- Platform implementations — Node (all four algorithms via
node:zlibwith per-chunk flush for SSE), Bun (Bun.gzipSync/Bun.deflateSync/Bun.zstdCompressSync+node:zlibbrotli + extendedCompressionStream), Deno (CompressionStreamgzip/deflate +node:zlibcompatibility for br/zstd), Web fallback (gzip/deflate viaCompressionStream). HttpServerResponse.removeHeader— new combinator needed forContent-Encoding: identitystripping andcontent-lengthremoval on streaming compression.- Tests — conformance matrix under web handler (negotiation, q-values, wildcards, malformed headers, minSize, content types, opt-outs,
no-transform, status codes, HEAD, streaming, ETag weakening, Vary merging, cookies); platform integration tests for Node and Deno (advertised algorithms, exactContent-Length, br/zstd round-trips, file responses, end-to-end server, SSE incremental flush deadlock). - Changeset — patches
effect,@effect/platform-node,@effect/platform-bun,@effect/platform-deno.
The Bun platform implementation is untested in CI (no Bun test infrastructure in this repo). The PR description notes it was verified locally — no issue, just recording the gap for posterity.
Note: 1 inline comment(s) dropped because they did not anchor to lines inside the PR diff:
packages/platform-node/src/NodeHttpPlatform.ts:1439-1440(RIGHT) — line 1440 (RIGHT) is not inside a diff hunk
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
|
(The one inline comment about In the
|
| return ServerResponse.setBody( | ||
| response, | ||
| HttpBody.uint8Array(compressSync(body.body, algorithm, options), body.contentType) | ||
| return ServerResponse.stream( |
There was a problem hiding this comment.
I think we should use a .raw response for this case
There was a problem hiding this comment.
Addressed in efc8462: the Uint8Array case now returns ServerResponse.raw around Readable.from([body.body]).pipe(compressTransform(...)), while preserving the response metadata and dropping Content-Length.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- Removed
compressSyncfrommakeCompression— the sync one-shot path is gone; all compression (includingUint8Arraybodies) now goes through the streaming path, droppingContent-Lengthon every compressed response. This simplifies the API and avoids the maintenance cost of dual code paths. - Inlined
CompressionMiddlewareOptionsas an anonymous object type on thecompressionfunction parameter, keeping the public interface self-contained. - Added identity guard in
wrapCompression(compressed === response) — prevents stampingcontent-encoding,vary, and ETag headers on responses thatcompressResponsereturned unchanged (e.g.Rawbodies with null content that couldn't be transformed). - Added bidirectional error forwarding in Node
Rawbody case —transform.on("error", (cause) => readable.destroy(cause))now complements the existing source→transform direction, fixing the gap noted in the initial review. - Added
ZSTD_e_flushand flush control to Deno's zstd path — mirrors Node's SSE-friendly per-chunk flush behavior; includesnode.d.tstype updates. - Removed
BROTLI_PARAM_SIZE_HINTfrombrotliParams— the size hint was only relevant for the removed sync one-shot path and has no place in the streaming approach. - Removed redundant
|\*alternative fromacceptMemberregex — the*character is already present in the character class; simplifies without changing behavior. - Moved module-level constants (
defaultAlgorithms,defaultLevels,noTransformRegex,contentLengthHeader) below thecompressionfunction. - Updated tests — assertion changes from exact
Content-Lengthtonullfor all compressed responses; new test"does not stamp headers when a Raw body cannot be transformed"exercises the identity guard; cleaned up test option types.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- Node
Uint8Arraycompression now usesServerResponse.raw— thecompressResponseUint8Arraycase was changed fromServerResponse.stream(NodeStream.pipeThroughDuplex(Stream.succeed(body.body), ...), ...)toServerResponse.raw(Readable.from([body.body]).pipe(compressTransform(...)), ...). This avoids wrapping a single-chunk body in an EffectStreamonly to pipe it through a duplex transform, instead creating a NodeReadabledirectly and piping it into the compression transform. The response metadata (status, headers, cookies,Content-Lengthremoval) is unchanged, andwrapCompressioncontinues to layer onContent-Encoding,Vary, and ETag weakening. Error propagation is handled by.pipe()— the source is a trivial in-memory readable that does not need explicit bidirectional forwarding.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- Made
compressiona required member ofHttpPlatform.make— thecompressionparameter onHttpPlatform.makeand thecompressionfield on theHttpPlatformservice type were changed from optional (compression?: Compression | undefined) to required (compression: Compression). The?? compressionWeb()fallback insidemakewas removed, and thelayerfactory now explicitly passescompression: compressionWeb(). - Updated test stubs in
HttpStaticServerConditional.test.ts— threeHttpPlatform.of()calls now includecompression: stubCompression(a no-op stub with an empty algorithm set and identitycompressResponse).
The middleware's runtime fallback (Context.getOrUndefined + ?? compressionWebWrapped()) is unchanged — it still handles the case where HttpPlatform is absent from the context.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
- Share one CompressionStream transform helper across web, Bun, and Deno - Rebuild compressed responses via setBody/removeHeader instead of hand-spreading response fields at five sites - Merge the Node Uint8Array and Raw cases, drop the Blob copy for one-chunk bodies, and replace lazy singletons with plain consts - Hoist per-request work in the middleware and trim dead declarations from the Deno node compat shim Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
compressionWeb/compressionWebWrappednow module-level constants — changed from lazily-initialized closures to eager constants sincemakeCompressionandwrapCompressionreturn plain objects with no runtime state dependency.- Shared
compressionStreamutility — extracted theCompressionStream-backed transform as a reusable helper, exported onHttpPlatformfor platform use (Bun, Deno). compressedBody/setBodyWithoutLength/streamBody/singleChunkStreamhelpers — factored out repeatedsetBody → removeHeader("content-length")patterns, replacing manual response reconstruction withHttpBodyconstructors that preserve all response fields viasetBody.- Node
Uint8Arraycase merged intoRaw—Uint8Arraybodies now createReadable.from([body.body])and fall through to the unifiedRawhandling with bidirectional error forwarding (harmless for in-memory readables). - Deno/Bun gzip/deflate now use
Platform.compressionStream— switched from inlinestream.pipeThrough(new CompressionStream(...))to the shared utility. withVaryextracted as module-level function — moved from closure-local to module scope for clarity.accept-encodingread inside pre-response handler — reads the header from the handler's request parameter instead of capturing it in the outer closure; behaviorally identical.node.d.tstype cleanup — removedZlibCompressOptions(no longer needed), inlinedflushintoBrotliOptions.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- Made
HttpPlatformrequired in the compression middleware —Context.getUnsafereplaces thegetOrUndefined+compressionWebWrappedfallback, addingHttpPlatformto the middleware's required dependency signature. Every call path that uses the middleware (viaHttpServer.layerServices,HttpPlatform.layer, or a platform-specific layer) already providesHttpPlatform, so the requirement is always satisfied.compressionWebWrappedis removed as dead code. - Renamed
compressionStream→compressionTransformWeb— makes the Web-stream-based nature explicit. All callers updated (Bun, Deno, internalcompressionWeb). - Renamed
makeCompression→makeCompressionWeb— same motivation. All callers updated (Bun, Deno). - Test harness improvements —
makeHandlernow usestoWebHandlerLayerwithHttpServer.layerServicesfor the default path (withafterAlldisposal via adisposersarray), andtoWebHandlerWithfor explicit-context negotiation tests.platformContext/negotiationPlatformContexthelpers properly typed viaHttpPlatform.of()withoutas unknown ascasts.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- HEAD request body cancellation in Node — added
cancelResponseBodythat destroysReadablestreams forRawbodies when handling HEAD requests, preventing file descriptor leaks when compression middleware runs on file responses with a HEAD verb. - Deno transform close cleanup — added
transform.on("close", () => source.destroy())to the br/zstd compression path, ensuring the sourceReadableis cleaned up when the compression transform closes (complementing the existing source→transform error forwarding). - Deno
node.d.tstype updates — addedon("close", ...)anddestroy(cause?)to theReadabletype declarations to support the close cleanup code. - Bun compression tests in CI — new
BunHttpCompression.test.tscovering one-shot, stream, andBun.filecompression round-trips for all four algorithms; backed by atest-bunCI job under Bun 1.3.13 and a@effect/platform-bunvitest project entry. - BREACH security documentation — added a Security section to the
compressionJSDoc explaining BREACH-style attack risks and mitigations (Content-Encoding: identity,Cache-Control: no-transform,compressiblepredicate). - HEAD close test in Node compression suite — new test verifies that compressed file bodies are properly closed (file descriptor released) when an HTTP HEAD request is served.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
compressResponsemade effectful — theCompression.compressResponsemethod and thetransformfunction now returnEffect.Effect<HttpServerResponse>instead ofHttpServerResponse, enabling asynchronous one-shot compression.wrapCompressionmaps over the effect to layer onContent-Encoding,Vary, and ETag weakening.- Async one-shot
Uint8Arraycompression — Node, Bun, and Deno platforms usenode:zlibcallback APIs (Zlib.gzip,Zlib.deflate,Zlib.brotliCompress,Zlib.zstdCompress) wrapped inEffect.callbackforUint8Arraybodies, producing an exactContent-Lengthfrom the compressed result. Stream and raw bodies remain on the existing streaming transform paths. makeCompressionWebwraps sync returns inEffect.succeed— the Web-stream-based compression helper now conforms to the effectfulcompressResponsesignature;compressionWebremains a module-level constant.node.d.tsextended — addedZlibCompressOptions,CompressCallback, and async callback API declarations (gzip,deflate,brotliCompress,zstdCompress) for Deno's TypeScript build compatibility.- Tests assert exact
Content-Length— platform-level one-shot compression tests now verifycompressed.byteLength.toString()instead ofnull, andzstdSupporteddetection usestypeof Zlib.zstdCompressconsistently across all platforms. stubCompressionand test fixtures updated —compressResponseon stubs usesEffect.succeed;HttpStaticServerConditionaltest fixtures pass the new requiredcompressionfield.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
The one-shot zlib compression, streaming transforms, and algorithm detection were copy-pasted across the Node, Bun, and Deno platforms. Move them to a shared NodeHttpCompression module, which also removes the Deno-local node:zlib/node:stream type shims. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior Pullfrog review:
- Extracted shared
NodeHttpCompressionmodule in@effect/platform-node-shared— the async one-shotUint8Arraycompression, streamingDuplex/WebReadableStreamtransforms, and zstd runtime detection that were duplicated across Node, Bun, and Deno now live in a single source file. Bun and Deno delegateUint8Arrayhandling toNodeHttpCompression.make()wrapped around their existingmakeCompressionWebstreaming fallback; Node delegatesStream/Rawhandling to its own fallback whilemake()handlesUint8Array. - Removed Deno's
internal/node.d.ts— thenode:zlibandnode:streamtype stubs are no longer needed sinceNodeHttpCompression.tslives inplatform-node-sharedwhich has native Node type support. - Updated barrel exports —
platform-node-shared/src/index.tsnow exportsNodeHttpCompression.
DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) ｜ 𝕏
1 participant
Implements the response compression spec for the v4 HttpServer.
Shared middleware (
effect/unstable/http)HttpMiddleware.compression(options?)withalgorithms(default["br", "gzip", "deflate"], zstd opt-in),minSize(default 1024),compressiblepredicate, and per-algorithmlevels(gzip 6, deflate 6, br 4, zstd 3).*matches unlisted codings, malformed headers are treated as absent, and no acceptable coding sends identity with 200 (never 406).Content-Encoding(withidentitystripped as the opt-out),Cache-Control: no-transform, non-compressible or absent content type, known length belowminSize, Empty/FormData bodies. HEAD requests mirror GET headers.Vary: Accept-Encodingis appended (no duplicates,Vary: *untouched) on every eligible response, including ones skipped by negotiation orminSize.cors) so it composes withHttpEffect.toWebHandler,HttpRouter.serve, and the platform servers.HttpPlatform.compressionprimitivecompression: { algorithms, compressResponse }member on the service;HttpPlatform.makewraps the platform transform with shared header logic (Content-Encoding,Vary, strong ETags weakened toW/...).Uint8Arraybodies on Node.js, Bun, and Deno use the asynchronousnode:zlibcallback APIs throughEffect.callback. The exact compressedContent-Lengthcomes fromresult.byteLength; there is no sync path or size-based switch to streaming.StreamandRawbodies remain streaming transforms and dropContent-Length.node:zlib(zstd gated on runtime support); async one-shot APIs forUint8Arraybodies and zlib transforms with per-chunk flush forStream/Rawbodies.node:zlibasync implementations for one-shot bodies and extendedCompressionStreamfor streaming, includingBunFileresponses. NoBun.*Syncone-shot branch is used.node:zlibcompatibility one-shot APIs; streaming gzip/deflate useCompressionStream, while br/zstd usenode:zlibcompatibility streams.CompressionStream; byte arrays are necessarily sent through a single-chunk stream becausenode:zlibis unavailable.Tests
Content-Lengthfor async one-shot bodies and continue to assert absentContent-Lengthfor stream/file responses.Validation:
pnpm lint-fixpnpm checkpnpm --dir packages/platform-bun checkpnpm --dir packages/platform-deno checkCloses EFF-332
Closes EFF-352
🤖 Generated with Claude Code