Skip to content

http-zstd

Pre-release
Pre-release

Choose a tag to compare

@eilandert eilandert released this 18 May 20:32
· 341 commits to master since this release

What's Changed

https://deb.myguard.nl/2026/05/zstd-nginx-module-what-it-does-bugs-fixed/

This release is a test release, although we think code is now stable and production worthy it has to mature too. Please leave your feedback.

New Directives & Features

  • zstd_long: Enable long-distance matching to achieve ~2:1 compression improvement on large repetitive bodies (concatenated JSON, HTML boilerplate, logs) with configurable memory ceiling via zstd_window_log.
  • zstd_bypass: Per-request compression opt-out via nginx predicates for BREACH mitigation (exclude endpoints mixing secrets with user-controlled input; requires application-level defenses).
  • zstd_static startup warning: Warn operators at nginx startup if zstd_static enabled without gzip_vary to prevent CDN cache collision issues.
  • Comprehensive debug logging: Added ngx_log_debug diagnostic logging for all 8 header-filter rejection reasons, buffer decisions, compression state, and emit decisions (zero cost in release builds, visible with error_log debug).
  • zstd_max_length on chunked: Extended zstd_max_length enforcement to track running input total on chunked responses (no Content-Length) to prevent DoS from unbounded upstreams.

Added optimisations:

  • Per-request context reuse done right. Each request gets its own zstd context, but the underlying allocation is reused across requests in a worker via reset — correctness and speed.
  • Modern single-call streaming API. Replaced three legacy streaming calls and a deprecated init API with one ZSTD_compressStream2(), and dropped a custom allocator that added nothing.
  • Pledged source size. When Content-Length is known, the exact size is pledged to zstd up front, producing a more compact frame header and slightly better speed and ratio at zero cost.
  • Output buffers default to ZSTD_CStreamOutSize(). The encoder’s own recommended granularity (~128 KB), so zstd never has to fragment a block across calls. Two buffers, so one fills while the other is in flight.
  • Hot-path loc-conf hoist. The location config is resolved once per request instead of once per inner-loop iteration — measurable on large responses.
  • Single-division ratio math for $zstd_ratio instead of dividing twice.
  • Skip a redundant loop iteration for an empty terminal buffer in the add-data path.
  • New operator capabilities: zstd_max_length (cap huge responses), zstd_window_log (bound per-request memory — predictable RSS under load), zstd_long (long-distance matching for big repetitive bodies), zstd_bypass (the BREACH lever), and $zstd_bytes_in / $zstd_bytes_out log variables for real observability.

Fixed bugs:

  • Silent truncation at 131072 bytes. Any response larger than zstd’s internal stream buffer (ZSTD_CStreamInSize, exactly 128 KB) was chopped off at that boundary. Big CSS bundles and JavaScript files simply ended mid-stream. This is the bug that makes a site look fine on the homepage and shatter the moment a real asset loads.
  • Terminal frame never emitted on empty end. When the final compression call produced zero output bytes (it happens — the encoder had already flushed everything), the module forgot to write zstd’s end-of-frame marker. Decoders saw a stream that just stopped and rejected it.
  • Truncation/abort with proxy_buffering off (the “bug B” saga). This was the production incident that took down deb.myguard.nl‘s wp-admin. With unbuffered proxying (the FastCGI/PHP path), the upstream forces a flush around a chunk that zstd buffers internally. The directive logic never told libzstd to actually flush, so it sat on the bytes forever — manifesting as a truncated response and a worker spinning at 100% CPU in a livelock. The fix has two halves: map a pending flush to ZSTD_e_flush, and clear the flush state when the encoder reports it’s fully drained so the loop can terminate.
  • NULL-deref / worker SIGSEGV on multi-buffer chunked responses. On any response big enough to need more than one output buffer (chunked or no-Content-Length is the common trigger), a recycled buffer pointer was dereferenced after being invalidated — a clean worker segfault that only appeared past the first ~128 KB.
  • 100% CPU infinite loop. A separate flush state-machine defect (PR #23 + #49 class) where the inner compression loop never reached a termination condition, pinning a core until the request timed out.
  • ZSTD_CDict leaked on every config reload. If you used a compression dictionary, every nginx -s reload leaked the entire dictionary. On a busy box that reloads config often, that’s a steady climb to OOM.
  • Cleanup handler registered with the wrong size. The dictionary cleanup used a non-zero size in ngx_pool_cleanup_add(), which subtly mismanaged the cleanup bookkeeping. Fixed to pass size=0 as the API intends.
  • Shared compression context across requests. The original reused one zstd context worker-wide. Overlapping requests in a single worker could stomp each other’s compression state — intermittent corruption that’s almost impossible to reproduce on demand. Now every request gets its own context, attached to the request’s cleanup chain.
  • Version-specific dictionary init error handling. On older libzstd, the dictionary init path used a different API whose error return wasn’t checked correctly, so a failed init could proceed as if it had succeeded.
  • Buffer overflow appending the .zst extension. The static module built the .zst path without reserving enough room, so a long enough request URI could write past the buffer. Classic overflow, now bounded and regression-tested with a 2000-character URI.
  • Integer overflow in the compression-ratio calculation. The $zstd_ratio math could overflow on large responses, producing garbage stats. Reworked with 64-bit scaling.
  • Reject oversized numeric directives (INT_MAX hardening). zstd_window_log and zstd_target_cblock_size are narrowed to int before being handed to libzstd. A configured value above INT_MAX would silently truncate (possibly to a negative). Now rejected at config load with a clear error.
  • Dictionary-file size cap. A 10 MB ceiling on zstd_dict_file prevents a memory-exhaustion denial of service via a giant dictionary.
  • BREACH containment lever (zstd_bypass). A new per-request opt-out so operators can serve identity (uncompressed) on endpoints that mix a secret with attacker-influenced reflected input — the precondition for a BREACH-style attack — without splitting the location config.
  • Unbounded input cap for chunked responses. A streaming response has no Content-Length, so the size check was skipped and a runaway upstream could feed the compressor forever. zstd_max_length is now enforced against the running input total too.
  • The -fPIC linking failure. The config preferred a static libzstd.a that lacks position-independent code and cannot link into a shared dynamic module. Now prefers dynamic linking with a pkg-config fallback.
  • Global CFLAGS mutation. The static module’s config leaked compiler flags into the global build, polluting other modules. Removed.
    Cross-platform portability. Build logic hardened for Linux, the BSDs, RHEL/Rocky, and Gentoo — different pkg-config tools, different library locations.
  • Cross-compilation support. Set ngx_feature_run=no so the feature probe doesn’t try to execute a test binary (impossible when cross-compiling).
  • Doubled source path. A bug in the addon config concatenated the source directory twice; fixed by scoping ngx_addon_dir per sub-config.
  • Typo: SAVED_CC_TAST_FLAGS. A real, shipped typo that silently failed to restore test flags. One character; genuinely broke the build path.
  • Filter ordering vs. brotli. If both zstd and brotli are enabled, zstd must run first or the ordering produces wrong results. Now enforced.
  • RPATH embedding documented for custom ZSTD_LIB paths, plus advisories when building against libzstd older than 1.4.0 or 1.5.0 (missing APIs).
  • Non-redistributable test fixture replaced with a BSD-2-Clause original, so the whole repo is cleanly licensed.
  • Accept-Encoding parsing was wrong. The header matcher could mis-detect zstd as a substring of another token. Rewritten to tokenize properly.
  • Quality values (q=) ignored. Accept-Encoding: zstd;q=0 means “do NOT send me zstd.” The module compressed anyway. Now it honors RFC 7231 qvalues, including the strict 0–1 range.
  • Content-Type detection used the wrong filename in the static module, so .zst files could get the wrong MIME type. Fixed to use the original (pre-.zst) name.
  • Only 200/403/404 were compressed. Other perfectly compressible 2xx responses (201, 206-not-applicable aside) were skipped. Broadened correctly.
  • 204 and 205 were compressed. Bodyless responses must not carry a Content-Encoding. Now excluded.
  • Accept-Ranges not cleared on compressed static files. Byte ranges into a .zst file are meaningless (offsets don’t map to the original). Now cleared per RFC 9110.
  • Default compression level changed 1 → 3. Level 1 left obvious ratio on the table for negligible CPU savings; 3 is the sane default.
  • Missing gzip_vary warnings. If zstd is enabled but gzip_vary is off, proxies and CDNs can cache a compressed response and serve it to a client that can’t read it. Both the filter and static modules now warn loudly at config load. (We added the static-module half this week.)
  • Two no-op / unreachable code blocks removed during audit, plus a cluster of eight smaller correctness fixes batched from the first full codebase audit (dead code, deduplication, minor bugs).
  • Skip-reason and diagnostic debug logging. “Why isn’t my response compressed?” was previously undiagnosable without a rebuild. Every eligibility-rejection path, the buffer-acquire decision, and the compression-context setup now emit structured debug logs (zero cost in release builds, visible under error_log debug). The permanent emit-decision probe for the truncation bug class is part of this group too.

Added CI pipeline:

  • Build & Test workflow with NGINX 1.31.0 and Angie 1.15.4, regression tests for every bug in the gitlog.
  • ASAN + UBSAN build and soak
  • Monthly valgrind Memcheck soak
  • CodeQL, security scanners, and fuzzing

Full Changelog: https://github.com/eilandert/zstd-nginx-module/commits/0.90