Repository navigation
zstd, gzhttp: RFC 9842 dictionary-compressed zstd (dcz) #1285
oddharsh
started this conversation in
Ideas/Features
Replies: 1 comment
|
Thanks for the heads up. Let me read through the RFC to understand the implications. At first sight, it seems a bit like a (complex) solution looking for a problem. But I can see possible benefits - of course assuming clients will support it. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
RFC 9842 (Compression Dictionary Transport, September 2025) defines a
dczcontent encoding. A dcz body is a 40-byte zstd skippable frame carrying the SHA-256 of a raw dictionary, followed by a zstd stream compressed with that dictionary.I'd like to generate and test dictionaries for my website and its assets at the library level, alongside compression. Cloudflare's shared dictionaries beta passes dcz through from the origin, so the origin builds the dictionaries and the deltas. In Go, that means building them with
dict.BuildRawDict, checking them, and serving them as dcz, without shelling out to the zstd CLI.I checked that loop against master (62d757c) with a small round-trip test, and it works today with no new API:
dict.BuildRawDict(orbuilddict -format raw) builds the raw dictionary type that RFC 9842 specifies for dcz.(&Header{Skippable: true, SkippableID: 0xe, SkippableSize: 32}).AppendTo(nil)returns the RFC's 8 magic bytes,5e 2a 4d 18 20 00 00 00. The caller appends the hash.WithEncoderDictRaw(0, dict)writes no dictionary ID, the same setupExampleWithEncoderDictRawuses for--patch-from.WithDecoderDictRaw(0, dict)decodes the whole dcz body as is, throughDecodeAlland the streaming reader, since the decoder skips the skippable frame.zstd -d -D, and Go decodes bodies made withzstd -Dandzstd --patch-from.On real data, I used this repo's README.md from 2026-02-18 as the dictionary for today's (60,291 B). Plain zstd from this package at the default level gives 15,705 B. dcz gives 5,435 B at the default level and 2,842 B at
SpeedBestCompression, while the same dcz body built withzstd -19 -Dis 2,613 B.The gap is on the HTTP side, where gzhttp negotiates only gzip and zstd. Today a server has to do dcz in its own handler. It checks
Accept-Encodingfordcz, looks up the dictionary whose hash is inAvailable-Dictionary, and setsContent-Encoding: dczplusVary: accept-encoding, available-dictionary. gzhttp passes those responses through unchanged, since it skips any response that already has aContent-Encoding. Each user would build the same negotiation though, and gzhttp'sMinSizeandContentTypesrules don't apply to those responses. Cloudflare's setup guide suggests doing this in existing compression middleware, which is the slot gzhttp fills for Go servers.Two shapes I could see, smallest first:
Exampleand test, in zstd or dict, that build a raw dictionary, write a dcz body with it, and read it back using the existing API. No gzhttp change.Would either fit the project? If 1, I'm happy to send it as a PR. If 2, what shape would you want for the dictionary lookup?
Side note: if the Transport ever offered dcz, its 32 MiB window cap would reject some windows RFC 9842 requires (at least 8 MB or 1.25x the dictionary, whichever is greater, up to 128 MB).
All reactions