-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
Is your feature request related to a problem? Please describe.
Yes. When using ZSTD_compressStream2 to compress a single input buffer (with ZSTD_e_end enablingZSTD_c_stableOutBuffer parameter), I encountered the Destination buffer is too small error. This occurred even though my output buffer size was equal to the input buffer size (which should be smaller than ZSTD_compressBound(input_size) for small inputs). However, when I first call ZSTD_compressStream2 with ZSTD_e_continue (potentially requiring multiple calls for flushing) and then in ZSTD_e_end mode, the compressed output size is smaller than the input size. This suggests that the upfront buffer size check triggered by ZSTD_e_end + ZSTD_c_stableOutBuffer might be overly conservative for very small buffers where compression cannot be achieved.
Describe the solution you'd like
I'd like ZSTD_compressStream2 called directly with ZSTD_e_end (enabling ZSTD_c_stableOutBuffer ) to maintain the streaming behavior, incrementally writing data and only returning the dstSize_tooSmall error when the output buffer is truly exhausted during the compression process, not if the final compressed size could potentially exceed the buffer (as predicted by ZSTD_compressBound for very small inputs). Essentially, defer the dstSize_tooSmall check until compression actually runs out of output space, similar to how the multi-call (ZSTD_e_continue followed by ZSTD_e_end) approach works.
Describe alternatives you've considered
Additional context
My test string for above case is "test test test by test test in unit test. test test repetition."