Be notified of new releases
Create your free GitHub account today to subscribe to this repository for new releases and build software alongside 28 million developers.Sign up
Zstandard v1.3.5 is a maintenance release focused on dictionary compression performance.
Compression is generally associated with the act of willingly requesting the compression of some large source. However, within datacenters, compression brings its best benefits when completed transparently. In such scenario, it's actually very common to compress a large number of very small blobs (individual messages in a stream or log, or records in a cache or datastore, etc.). Dictionary compression is a great tool for these use cases.
This release makes dictionary compression significantly faster for these situations, when compressing small to very small data (inputs up to ~16 KB).
The above image plots the compression speeds at different input sizes for
zstd v1.3.4 (red) and v1.3.5 (green), at levels 1, 3, 9, and 18.
The benchmark data was gathered on an
Intel Xeon CPU E5-2680 v4 @ 2.40GHz. The benchmark was compiled with
clang-7.0, with the flags
-O3 -march=native -mtune=native -DNDEBUG. The file used in the results shown here is the
osdb file from the Silesia corpus, cut into small blocks. It was selected because it performed roughly in the middle of the pack among the Silesia files.
The new version saves substantial initialization time, which is increasingly important as the average size to compress becomes smaller. The impact is even more perceptible at higher levels, where initialization costs are higher. For larger inputs, performance remain similar.
Users can expect to measure substantial speed improvements for inputs smaller than 8 KB, and up to 32 KB depending on the context. The expected speed-up ranges from none (large, incompressible blobs) to many times faster (small, highly compressible inputs). Real world examples up to 15x have been observed.
Other noticeable improvements
The compression levels have been slightly adjusted, taking into consideration the higher top speed of level 1 since v1.3.4, and making level 19 a substantially stronger compression level while preserving the
8 MB window size limit, hence keeping an acceptable memory budget for decompression.
It's also possible to select the content of
libzstd by modifying macro values at compilation time. By default,
libzstd contains everything, but its size can be made substantially smaller by removing support for the dictionary builder, or legacy formats, or deprecated functions. It's even possible to build a compression-only or a decompression-only library.
Detailed changes list
- perf: much faster dictionary compression, by @felixhandte
- perf: small quality improvement for dictionary generation, by @terrelln
- perf: improved high compression levels (notably level 19)
- mem : automatic memory release for long duration contexts
- cli : fix :
overlapLogcan be manually set
- cli : fix : decoding invalid lz4 frames
- api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln
- api : change : clarify
ZSTD_CCtx_resetParameters(), by @terrelln
- build: select custom
libzstdscope through control macros, by @GeorgeLu97
- build: OpenBSD support, by @bket
make allare compatible with
- doc : clarify
zstd_compression_format.md, updated for IETF RFC process
pzstdcompatible with reproducible compilation, by @lamby
zstd --list does not work with non-interactive tty.
This issue is fixed in