ZSTD_estimateCCtxSize() and ZSTD_compressSequences()

[encode84]

If allocate memory according to ZSTD_estimateCCtxSize() with level 3 or below, the ZSTD_compressSequences( ) will return error -64 (ZSTD_error_memory_allocation)
With level 4 and above it works fine. Is it expected behavior?

(Of course I'm using it in pair with ZSTD_initStaticCCtx())

If use ZSTD_compressSequences() with ZSTD_createCCtx() no issues what so ever.

[Cyan4973]

Hi Ilya,

the expectation is that ZSTD_estimateCCtxSize() returns a size which is suitable for the level specified and any level below it : https://github.com/facebook/zstd/blob/v1.5.5/lib/zstd.h#L1644 .

Therefore, if one requests a size for level 3, it will be fine for levels 1, 2 and 3, but there is no expectation for it to work for level 4.

Occasionally, it might be possible for a higher level to fit in, because, by chance, the technical parameters of this level happen to not require more memory, so it fits. But there is no guarantee, and as the technical definition of "levels" change overtime, this observation may also change across versions.
The technical definition of levels is hosted here : https://github.com/facebook/zstd/blob/dev/lib/compress/clevels.h .

[encode84]

Hello Yann,
The problem is that I pass the same compression level to all of these functions (via ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel), That's why I alarmed it.
ZSTD_estimateCCtxSize(3) memory will be not sufficient for compression with ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 3) + ZSTD_compressSequences(cctx, ...

[Cyan4973]

ZSTD_estimateCCtxSize() was meant to be used with ZSTD_compressCCtx().

ZSTD_compressSequences() is a completely different entry point, and I'm not sure how it would work in this case.
I'm not even sure if ZSTD_compressSequences() is compatible with ZSTD_initStaticCCtx()...

Since ZSTD_compressSequences() will compress the sequences that you provide, it completely skips the match searching stage, which means it doesn't even need memory for the search tables.
So it actually uses less memory ? Well for search certainly, but on the other hand, it needs more memory specifically to handle these externally provided sequences, and build some internal representation of it.
In short, its memory requirement is somewhat unrelated to compressionLevel.

I don't even know how much it needs, since this combination with ZSTD_initStaticCCtx() was never tested.

edit : one method to get an appropriate size for ZSTD_initStaticCCtx() could be this one :

• Allocate a normal ZSTD_CCtx* state (with full malloc capability)
• Run your scenario as wanted (using ZSTD_compressSequences())
• At the end, request ZSTD_sizeof_CCtx(cctx), this should return the state size
• Assuming everything ZSTD_compressSequences() needs can fit into the state size, this should be a large enough value for ZSTD_initStaticCCtx().

There is still a risk that ZSTD_compressSequences() also requires some dynamic memory sizing capability outside of the scope of ZSTD_initStaticCCtx(). In which case, no size will ever be large enough. The best way to know that would be to test it with some dummy very large state size.

[encode84]

Thank you, Yann!
It might be a good idea to add a special level for ZSTD_estimateCCtxSize(), e.g. level 255 or something to indicate entropy only operations for ZSTD_compressSequences()