Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Ability to make magic number optional #591
The frame header's 4 byte magic number header can be a substantial size of a full frame when doing dictionary compression on small data. In the case of the Firefox Mercurial repository, there are ~3M "chunks" that need compressed. That would produce ~12MB of zstd magic numbers in what is currently ~1800 MB of zlib-compressed data. That's ~0.67%, which feels high. In some cases, the 4 byte magic number is the difference between zstd (without dictionary compression) being smaller than zlib.
I know there is a block API that can be used to bypass the zstd frame header. But that requires a bit too much wheel reinvention for me. The frame header is generally useful and aside from the magic number I think it is reasonably designed in terms of not wasting space.
I'd like to float the idea for making the magic number optional in both the compressor and decompressor so advanced consumers (who care about size) can omit it while having what is a near fully-functional frame. If this were implemented, I imagine it would be built on top of
Are you receptive to this feature request?
I'm mostly worried about the complexity.
We want to keep the interface manageable.
Once it's in place, new feature, such as this request, will be easier to present.
I'm tentatively fine waiting until a
There's some news on this front.
This new API features
The API has a larger scope though, and answers the following 2 main issues :
Anyway, this is still early days, and we have ample time to collect feedbacks, and modify the interface when needs be. This API will remain in "experimental" section for some releases, and join "stable" status when deemed ready.
added a commit
Sep 28, 2017
We finally added this requested capability in
It can be accessed through new advanced API :
While the generation of such magic-less format is easy, it was more complex to support on the decoder side.
Without this instruction, the decoder will fail, as it expects the magic number.