zstd support

[ricea]

Issue for discussion of adding support for the zstd format to the API.

[ricea]

Currently libzstd is not linked into Chromium. It would add roughly 700KB to the binary size (maybe can be trimmed a bit). This would be a hard sell if the compression standard is the only user.

[evanstade]

It seems that you did manage to trim the size a bit as this change only added 38k. Was that just by dropping everything besides decompression, or were there other tricks you made use of?

On the storage team we are also interested in using zstd for compressing IndexedDB values before sending them over IPC (or storing as blobs). Adding the compression half of zstd appears to add another 200kB.

So that's at least two users of zstd.

[ricea]

We didn't do anything special in Chromium to reduce the size increase, except for only compiling in the decompression code. I think we were just lucky.

I would like to move this specification to WHATWG before we add new features, since it is now implemented in Firefox and Safari and we would like to have consensus.

That means it will probably be some months before there's movement on adding zstd.

[saschanaz]

I would like to move this specification to WHATWG before we add new features, since it is now implemented in Firefox and Safari and we would like to have consensus.

Why would it take that much to move to WHATWG?

[telgareith]

Note: Chrome/Chromium, Firefox, and Opera have agreed to add zstd to content-encoding options. Chrome is planning on General Availability in 118.

[Jedipedia]

Another note: Firefox now supports Content-Encoding: zstd as of version 126 but there's still no support for new DecompressionStream("zstd") in either Firefox or Chromium.

[xpl]

Please add it to the specification, as it seemingly blocks from implementing new DecompressionStream("zstd") in browsers:

https://issues.chromium.org/issues/348499728

[saschanaz]

#34 (comment)

Just as brotli, we'd be open to ship only DecompressionStream support for zstd, as @jesup reported 600KB size increase to ship the compression side of the library.

[ricea]

#34 (comment)

Just as brotli, we'd be open to ship only DecompressionStream support for zstd, as @jesup reported 600KB size increase to ship the compression side of the library.

Thanks. I think this will also be Blink's position.

[nektro]

Bun would be interested in supporting this since we already ship with zstd support internally for other APIs

[CraigglesO]

I heavily use DecompressionStream for gzip. I would love to see brotli and zstd support for this as reading data is drastically more useful/common in browsers than writing. Commenting to understand better the process here. Will this become a priority within the next year or so?

[ricea]

We need someone to write an implementation, a PR to update this specification, and web platform tests. Or three separate people, it doesn't matter.

At the last W3C TPAC we reached an agreement in principle to add this to the standard, someone just needs to do the work.

Unfortunately Chromium won't be able to ship this until there is some real-world usage, due to the binary size problem.

[saschanaz]

Did we? I think we only agreed about brotli.

[ricea]

@saschanaz My recollection is that we had a proper agreement for Brotli, and someone suggested we apply it to zstd too, and there was a general sense of agreement. However, if you think there's more to be done to get agreement for zstd, I'm happy to wait for that.

[saschanaz]

I remember (and per the meeting minutes) I checked again about zstd at the end of the meeting and everyone said brotli only... But maybe I'm wrong?

(if others are fine with it, Mozilla would be happy to consider exposing our ongoing zstd implementation)

[kenrussell]

Colleagues and I would like to see zstd support in DecompressionStream in support of the GPU texture "supercompression" format Basis Universal. This file format contains blocks of zstd-compressed coefficients. It is already used by libraries like Three.js and is a recommended texture format in glTF - both for texturing geometry in 3D web apps. Direct decompression support in the browser would reduce the size of WebAssembly modules which operate on these files. From our standpoint, compression support isn't needed. Thanks in advance for considering this; we're hoping this gets standardized.