Skip to content

Unexpected Destination buffer is too small error when compressing small buffers with ZSTD_e_end and ZSTD_c_stableOutBuffer #4458

@cghAndyReal

Description

@cghAndyReal

Is your feature request related to a problem? Please describe.
Yes. When using ZSTD_compressStream2 to compress a single input buffer (with ZSTD_e_end enablingZSTD_c_stableOutBuffer parameter), I encountered the Destination buffer is too small error. This occurred even though my output buffer size was equal to the input buffer size (which should be smaller than ZSTD_compressBound(input_size) for small inputs). However, when I first call ZSTD_compressStream2 with ZSTD_e_continue (potentially requiring multiple calls for flushing) and then in ZSTD_e_end mode, the compressed output size is smaller than the input size. This suggests that the upfront buffer size check triggered by ZSTD_e_end + ZSTD_c_stableOutBuffer might be overly conservative for very small buffers where compression cannot be achieved.

Describe the solution you'd like
I'd like ZSTD_compressStream2 called directly with ZSTD_e_end (enabling ZSTD_c_stableOutBuffer ) to ​maintain the streaming behavior, incrementally writing data and only returning the dstSize_tooSmall error ​when the output buffer is truly exhausted during the compression process, not if the final compressed size could potentially exceed the buffer (as predicted by ZSTD_compressBound for very small inputs). Essentially, defer the dstSize_tooSmall check until compression actually runs out of output space, similar to how the multi-call (ZSTD_e_continue followed by ZSTD_e_end) approach works.

Describe alternatives you've considered

Additional context
My test string for above case is "test test test by test test in unit test. test test repetition."

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions