Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BrotliCompressor.CopyToAsync buffer is not enough #249

Merged
merged 3 commits into from
Mar 15, 2024

Conversation

hadashiA
Copy link
Contributor

#243

BrotoliEncoder.Compress seems to return OperationState.DestinationTooSmall even when the destination buffer is short.

If the length of one BufferSegment is very large (more than the bufferSize parameter), finalState was DestinationTooSmall, the method would complete.
( In #243, MemoryStream is used).

@hadashiA
Copy link
Contributor Author

Here is the code to reproduce the issue.
#243 (comment)

I have written a slightly shortened version,

const string path1 = "/path/to/test.compressed";

var inputBytes = await File.ReadAllBytesAsync(path1);
System.Console.WriteLine($"Compressed file size (input) : {inputBytes.Length}");

var decompressor = new BrotliDecompressor();
var inputDecompressedBytes = decompressor.Decompress(inputBytes);
System.Console.WriteLine($"Decompressed size (input) : {inputDecompressedBytes.Length}");
var inputDeserialized = MemoryPackSerializer.Deserialize<TradeLog>(inputDecompressedBytes)!;
System.Console.WriteLine($"Decompressed data count (input) : {inputDeserialized.Data.Count}");
decompressor.Dispose();
System.Console.WriteLine("---------------");

var compressor = new BrotliCompressor(CompressionLevel.SmallestSize);
MemoryPackSerializer.Serialize(compressor, inputDeserialized);

// var buffer = new ArrayBufferWriter<byte>(inputBytes.Length);
// compressor.CopyTo(buffer);
var memoryStream = new MemoryStream(inputBytes.Length);
await compressor.CopyToAsync(memoryStream);
memoryStream.Flush();
memoryStream.Seek(0, SeekOrigin.Begin);
// System.Console.WriteLine($"Compressed size (serialized) : {buffer.WrittenSpan.Length}");
System.Console.WriteLine($"Compressed size (serialized) : {memoryStream.Length}");
compressor.Dispose();

var decompressor2 = new BrotliDecompressor();
var decompressedBytes = decompressor.Decompress(memoryStream.ToArray());
System.Console.WriteLine($"Decompressed size (serialized) : {decompressedBytes.Length}");
var deserialized = MemoryPackSerializer.Deserialize<TradeLog>(decompressedBytes)!;
System.Console.WriteLine($"Decompressed data count (serialized) : {deserialized.Data.Count}");
decompressor2.Dispose();

test.compressed data is available from the above comment.

@@ -174,10 +174,18 @@ public async ValueTask CopyToAsync(Stream stream, int bufferSize = 65535, Cancel
}

// call BrotliEncoderOperation.Finish
var finalStatus = encoder.Compress(ReadOnlySpan<byte>.Empty, buffer, out var consumed, out var written, isFinalBlock: true);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

スクリーンショット 2024-03-15 14 36 43

@hadashiA hadashiA merged commit f1fc4fc into main Mar 15, 2024
2 checks passed
@hadashiA hadashiA deleted the hadashiA/fix-brotli-encoder branch March 15, 2024 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant