How to compress extremely small amounts of data using zstd?

[windming-hub]

Hello, when I use zstd dictionary compression to compress 32Byte data, even the data are all the same, and I remove the magic, dictID and checksum in the result, the final compressed result is still 12byte. I found that only 1byte is a reference to the real data. Others are redundant metadata. Is there any way to reduce these redundant metadata?

[Cyan4973]

Probably not.
Even dictionary compression has limits. It's designed for inputs in the ~1 KB range, though it's still good enough for ~100 bytes input. But below that point, and especially below 50 bytes, fixed costs matter more and more and can dominate.

Even after doing all the things you mention, from the top of my mind, there is still :

• frame header : 2 bytes minimum
• block header : 3 bytes
• literals header : 1 byte (assuming it's empty)
• sequences description header : 1 byte (assuming it uses the dictionary of the fixed distribution)
• then the bitstream, which must contain the offset, the match length and the literal length (even if it's zero), plus the additional bits (mostly for offset).

That last part is a bit less clear to describe precisely. And it really depends on the produced sequence, here expected to be a single clean match reference into the dictionary. Assuming all this, I get the same result as you (for a 32-bytes reference).

We used to have a block format that could have helped a bit for this scenario (it was focused on small data),
as it would have got rid of the first 2 components. But actually, it merely transfers the responsibility somewhere else, as the information they contain (compressed and decompressed size notably) must still be represented somewhere.
Unfortunately, it's also annoying to maintain, because it has tons of limitations, and despite documentation, it is frequently misused into scenarios that it's not meant to serve. So we are moving away from this (experimental) interface, and it will likely be removed in some future version.

[windming-hub]

Probably not. Even dictionary compression has limits. It's designed for inputs in the ~1 KB range, though it's still good enough for ~100 bytes input. But below that point, and especially below 50 bytes, fixed costs matter more and more and can dominate.

Even after doing all the things you mention, from the top of my mind, there is still :

• frame header : 2 bytes minimum
• block header : 3 bytes
• literals header : 1 byte (assuming it's empty)
• sequences description header : 1 byte (assuming it uses the dictionary of the fixed distribution)
• then the bitstream, which must contain the offset, the match length and the literal length (even if it's zero), plus the additional bits (mostly for offset).

That last part is a bit less clear to describe precisely. And it really depends on the produced sequence, here expected to be a single clean match reference into the dictionary. Assuming all this, I get the same result as you (for a 32-bytes reference).

We used to have a block format that could have helped a bit for this scenario (it was focused on small data), as it would have got rid of the first 2 components. But actually, it merely transfers the responsibility somewhere else, as the information they contain (compressed and decompressed size notably) must still be represented somewhere. Unfortunately, it's also annoying to maintain, because it has tons of limitations, and despite documentation, it is frequently misused into scenarios that it's not meant to serve. So we are moving away from this (experimental) interface, and it will likely be removed in some future version.

So I wonder if you have a recommended method or algorithm to solve this problem? I would appreciate it.

[felixhandte]

Doing dictionary compression already requires external coordination between the compressor and decompressor. Removing the magic means you're already handling a non-conforming stream. If you are handling messages with extremely high redundancy / low entropy, perhaps you can just introduce your own custom scheme. If there's a limited set of possible messages that can be sent, you can define a list of all of them and then just communicate the index in the list.

Alternatively, you could try LZ4 compression, which similarly supports dictionary compression, but doesn't have any framing or block headers (as opposed to LZ4F, which does).

[windming-hub]

Thanks for your suggestion, best wishes!