[zstd_stream] Add Writer.Flush() #86

delthas · 2020-08-02T11:19:43Z

Add a fix for Writer.Close.
Add Writer.Flush()

See commits messages for detailed info.

Previously, if there was any remaining data to be written to the underlying Writer (which was almost always the case, since zstd tries to buffer blocks and frames as much as possible), and there was an underlying error when doing a Write on the underlying io.Writer, no error was returned from Close. This was an issue because clients would believe the stream was correctly written in its entirety even though the last bytes were not written. This fixes the issue by catching these errors, closing the zstd stream and returning the i/o error.

There are two levels of buffering in the library: - zstd buffers its data by not closing zstd blocks until a specific call to ZSTD_compressStream2 with ZSTD_e_flush or ZSTD_e_end is made - the Go wrapper buffers the input data to its own buffer Previously there was no way to flush these two buffers without closing the stream. This adds a way to flush these streams in order for any reader that wants to read was written to the stream until that point, to do so without waiting for the entire stream to be closed.

Viq111

Hello @delthas

Thanks for your contribution!
Sorry for the delay, I was out the last weeks but the code lgtm.

Thanks also for fixing the writer.Close() not returning errors on underlying writer failure

delthas added 2 commits August 2, 2020 13:11

Viq111 approved these changes Aug 17, 2020

View reviewed changes

Merge branch '1.x' into 1.x

ddae45c

Viq111 merged commit 4b8fdba into DataDog:1.x Aug 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[zstd_stream] Add Writer.Flush() #86

[zstd_stream] Add Writer.Flush() #86

delthas commented Aug 2, 2020

Viq111 left a comment

[zstd_stream] Add Writer.Flush() #86

[zstd_stream] Add Writer.Flush() #86

Conversation

delthas commented Aug 2, 2020

Viq111 left a comment

Choose a reason for hiding this comment