Consider using Zstandard compression for build cache artifacts

[lptr]

Facebook has a compression algorithm called Zstandard: https://facebook.github.io/zstd/

It seems to be much faster across the board than zlib, and it offers significant customizability wrt speed vs compression ratio.

The official implementation is native, but there are both JNI bindings for it and a pure Java implementation available.

We could use the JNI-based variant for platforms that are supported (Windows, macOS and various architectures on Linux), and fall back to the Java-only variant for unsupported ones.

The above graph seems to suggest that different environments could benefit from different compression ratios, i.e. local builds could use something that compresses faster, while CI builds could use something that takes longer to compress to a smaller size, so downloads become faster. See also #12810.

---

cc: @gradle/build-cache

[lptr]

Note that the Java-only variant in Aircompressor 0.16 doesn't support JVMs using big-endian byte order. Not sure if this has any significant impact for us.

[yogurtearl]

some thoughts:

• zstd support streaming so you could start decompression while bytes are streaming in from the remote cache.
  • https://raw.githack.com/facebook/zstd/release/doc/zstd_manual.html#Chapter7
• consider LZ4, which maybe a better tradeoff for some cases. https://lz4.github.io/lz4/
• both LZ4 and zstd support a "training mode" where you can create dictionaries for certain files types (.class, .json, .xml, etc ). The custom dictionaries can provide much better ratio and faster compress/decompress speeds.

[lptr]

All of these are good points, though compression speed is only a small part of caching speed. We know of lower hanging fruit for a number of typical use cases that would improve caching performance more than switching compression. (One example is clean builds in a non-ephemeral CI environment, where we currently delete everything first, then unpack exactly the same outputs byte-for-byte again, which is completely useless. See #10890)

This will become more interesting if we switch to a CAS-based cache protocol, where each output file will have to be stored individually. In that scenario training modes will likely have a significant advantage.

[lptr]

zstd support streaming so you could start decompression while bytes are streaming in from the remote cache.

FTR, this could also work with our current TAR+GZip-based protocol.

[Tapchicoma]

FYI: latest release of zstd brought quite nice performance improvements: https://github.com/facebook/zstd/releases/tag/v1.5.4

[lptr]

Nice, we might be in the for some new compression algorithms soon!

[kaeawc]

I've been using zstd to optimize Configuration Cache reuse on CI and its about 5x faster than standard tar gzip as a result - mostly in the speed to create and decompress the archive because of the sheer volume of files, not to actually make the overall size that much smaller. If its alright I might try to create a proof of concept to use zstd with jni and in-java fallback