-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ReadAsync doesn't return until output buffer filled or stream ends #32
Comments
I like the second option, we can increase the |
I think in that option |
Maybe something like (again, something I just quickly scratched together without testing): EnsureNotDisposed();
var output = new ZSTD_outBuffer_s { pos = 0, size = (nuint)buffer.Length };
while (true)
{
// Check if we already have some data available to decompress
lastDecompressResult = DecompressStream(ref output, buffer.Span);
if (output.pos > 0)
{
return (int)output.pos;
}
// Otherwise, read some more data
if (input.pos >= input.size)
{
int bytesRead;
if ((bytesRead = await innerStream.ReadAsync(inputBuffer, 0, inputBufferSize, cancellationToken)
.ConfigureAwait(false)) == 0)
{
if (checkEndOfStream && lastDecompressResult != 0)
{
throw new EndOfStreamException("Premature end of stream");
}
return 0;
}
input.size = (nuint)bytesRead;
input.pos = 0;
}
} |
|
True, but if I understand correctly it is still possible that the ZSTD_DCtx_s has consumed all compressed data from |
We can add this before ReadAsync: // frame is completely decoded so flush it before next read
if (lastDecompressResult == 0 && output.pos > 0) {
break;
} |
I'm afraid I don't see the purpose of checking if a frame has ended there. |
A frame ends at the flush/close of the compression stream, so a flush in compression will cause a flush in decompression. |
Flushing the compression would end the current block ( |
Should I make a pull request with my solution? |
Flush before if (output.pos > 0) {
break;
} |
I'm trying to use this library for adding compression to a network protocol, and the idea I had is to simply wrap the protocol in a ZSTD-compressed stream. Each time an end sends a message, it would flush out a compressed block. But I'm running into the problem that
DecompressionStream.ReadAsync
doesn't exit return the data it has already decompressed even if no more data is currently available on the inner stream. ReadAsync only exits onceoutput.pos >= output.size
, orinnerStream.ReadAsync
returns 0 (when the stream has ended). So there seems to be no way fetch all currently available data from a DecompressionStream when reading from a never-ending stream (unless you read one byte at a time from the DecompressionStream, but that would presumably be inefficient), and there seems to be no other way to access streaming decompression in this library without using unsafe code.I don't see any clear, clean way of having it check if there is any more available data (there is no generic DataAvailable on Stream, and no way to use timeout or cancellation with NetworkStream), so the only solutions I can think of would be to:
Have ReadAsync not await innerStream.ReadAsync if it already has some decompressed data, and instead possibly leave it in progress to be picked up by a subsequent call. Something like:
Though that's just something I quickly threw together, I haven't tested it, and Read would also need to be modified in order for mixed Sync/Async read to be safe, and I have no idea how to deal with that in Dispose.
Have
ReadAsync
never callinnerStream.ReadAsync
whenoutput.pos > 0
(which means it would callinnerStream.ReadAsync
at most once on each call). Would have the drawback that ReadAsync might not fill the output buffer even if there is more data available.Provide a thin, non-stream-based, safe wrapper around Decompressor.DecompressStream. Something like:
The text was updated successfully, but these errors were encountered: