Rewrite ChunkedMemoryStream #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
|
This is a benchmark review for experiment This pull request was cloned from Experiment configurationreview_config:
# User configuration for the review
# - benchmark - use the user config from the benchmark reviews
# - <value> - use the value directly
user_review_config:
enable_ai_review: true
enable_rule_comments: false
enable_complexity_comments: benchmark
enable_security_comments: benchmark
enable_tests_comments: benchmark
enable_comment_suggestions: benchmark
enable_pull_request_summary: benchmark
enable_review_guide: benchmark
enable_approvals: false
base_branches: [base-sha.*]
ai_review_config:
# The model responses to use for the experiment
# - benchmark - use the model responses from the benchmark reviews
# - llm - call the language model to generate responses
model_responses:
comments_model: benchmark
comment_validation_model: benchmark
comment_suggestion_model: benchmark
complexity_model: benchmark
security_model: benchmark
tests_model: benchmark
pull_request_summary_model: benchmark
review_guide_model: benchmark
overall_comments_model: benchmark
# The pull request dataset to run the experiment on
pull_request_dataset:
# CodeRabbit
- https://github.com/neerajkumar161/node-coveralls-integration/pull/5
- https://github.com/gunner95/vertx-rest/pull/1
- https://github.com/Altinn/altinn-access-management-frontend/pull/1427
- https://github.com/theMr17/github-notifier/pull/14
- https://github.com/bearycool11/AI_memory_Loops/pull/142
# Greptile
- https://github.com/gumloop/guMCP/pull/119
- https://github.com/autoblocksai/python-sdk/pull/335
- https://github.com/grepdemos/ImageSharp/pull/6
- https://github.com/grepdemos/server/pull/61
- https://github.com/websentry-ai/pipelines/pull/25
# Graphite
- https://github.com/KittyCAD/modeling-app/pull/6648
- https://github.com/KittyCAD/modeling-app/pull/6628
- https://github.com/Varedis-Org/AI-Test-Repo/pull/2
- https://github.com/deeep-network/bedrock/pull/198
- https://github.com/Metta-AI/metta/pull/277
# Copilot
- https://github.com/hmcts/rpx-xui-webapp/pull/4438
- https://github.com/ganchdev/quez/pull/104
- https://github.com/xbcsmith/ymlfxr/pull/13
- https://github.com/tinapayy/B-1N1T/pull/36
- https://github.com/coder/devcontainer-features/pull/6
# Questions to ask to label the review comments
review_comment_labels: []
# - label: correct
# question: Is this comment correct?
# Benchmark reviews generated by running
# python -m scripts.experiment benchmark <experiment_name>
benchmark_reviews: []
|
Author
Reviewer's GuideThe pull request rewrites File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Hellebore
commented
May 2, 2025
Author
Hellebore
left a comment
Hellebore
left a comment
There was a problem hiding this comment.
Hey @Hellebore - I've reviewed your changes - here's some feedback:
- The accessibility of
ChunkedMemoryStreamhas changed frominternal sealedtopublic; please confirm if this visibility change is intentional.
Here's what I looked at during the review
- 🟡 General issues: 3 issues found
- 🟢 Security: all looks good
- 🟡 Testing: 2 issues found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| int offset = 0; | ||
| int count = buffer.Length; | ||
| while (count > 0) | ||
| while (bytesToRead != 0 && this.currentChunk != this.memoryChunkBuffer.Length) |
Author
There was a problem hiding this comment.
issue (bug_risk): Incorrect chunk index comparison in Read method.
Comparing currentChunk to memoryChunkBuffer.Length (the total byte length) can overrun the chunks; use memoryChunkBuffer.ChunkCount instead.
| while (count > 0) | ||
| int bytesToWrite = (int)remaining; | ||
| int bytesWritten = 0; | ||
| while (bytesToWrite != 0 && this.currentChunk != this.memoryChunkBuffer.Length) |
Author
There was a problem hiding this comment.
issue (bug_risk): Incorrect chunk index comparison in Write method.
memoryChunkBuffer.Length is the total byte capacity, not the number of chunks. Use the actual chunk count (e.g. a ChunkCount property) to compare against currentChunk to avoid write errors.
Comment on lines
+182
to
+183
| MemoryMarshal.Write(this.singleByteBuffer, ref value); | ||
| this.Write(this.singleByteBuffer, 0, 1); |
Author
There was a problem hiding this comment.
suggestion: Simplify single-byte write in WriteByte method.
Use this.singleByteBuffer[0] = value instead of MemoryMarshal.Write for a clearer, potentially faster single-byte write.
Suggested change
| MemoryMarshal.Write(this.singleByteBuffer, ref value); | |
| this.Write(this.singleByteBuffer, 0, 1); | |
| this.singleByteBuffer[0] = value; | |
| this.Write(this.singleByteBuffer, 0, 1); |
| using MemoryStream ms = this.CreateTestStream(length); | ||
| using var cms = new ChunkedMemoryStream(this.allocator); | ||
| using MemoryStream ms = CreateTestStream(length); | ||
| using ChunkedMemoryStream cms = new(this.allocator); |
Author
There was a problem hiding this comment.
suggestion (testing): Consider testing interleaved read and write operations.
Add tests that interleave reads and writes without resetting the stream position to verify that position, currentChunk, and currentChunkIndex stay correct during mixed operations.
Suggested implementation:
[Fact]
public void MemoryStream_ReadWriteInterleavedTest()
{
// Arrange: Create a MemoryStream and a ChunkedMemoryStream instance.
int totalLength = DefaultSmallChunkSize * 16;
using MemoryStream ms = CreateTestStream(totalLength);
using ChunkedMemoryStream cms = new(this.allocator);
// Act Part 1: Write a portion of the data manually.
int splitPoint = totalLength / 2;
byte[] buffer = new byte[splitPoint];
// Read 'splitPoint' bytes from ms.
int readBytes = ms.Read(buffer, 0, splitPoint);
Assert.Equal(splitPoint, readBytes);
// Write those bytes into the chunked stream.
cms.Write(buffer, 0, splitPoint);
// Act Part 2: Write remaining data.
ms.CopyTo(cms);
// Without resetting the position, set the position to beginning.
cms.Position = 0;
byte[] resultBuffer = new byte[totalLength];
int bytesRead = cms.Read(resultBuffer, 0, resultBuffer.Length);
// Assert: Ensure all data was read correctly.
Assert.Equal(totalLength, bytesRead);
// Optional: If accessible, check internal tracking properties.
// For example, if cms exposes CurrentChunk and CurrentChunkIndex for testing:
// Assert.NotNull(cms.CurrentChunk);
// Assert.Equal(expectedChunkIndex, cms.CurrentChunkIndex);
}
[InlineData(DefaultSmallChunkSize * 16)]
public void MemoryStream_ReadByteBufferTest(int length)
{
using MemoryStream ms = CreateTestStream(length);
using ChunkedMemoryStream cms = new(this.allocator);
ms.CopyTo(cms);If the properties "CurrentChunk" and "CurrentChunkIndex" are internal or private, modify their access for testing purposes or use reflection to assert their values. Adjust the test as needed to fit the codebase's access patterns.
|
|
||
| public ChunkedMemoryStreamTests() => this.allocator = Configuration.Default.MemoryAllocator; | ||
|
|
||
| [Fact] |
Author
There was a problem hiding this comment.
suggestion (testing): Suggest testing ToArray() behavior with a non-zero position.
Add a test that sets the stream position to a non-zero offset before calling ToArray(), then verify the returned array content and that the stream’s position is restored.
Suggested implementation:
// Add the following new test method to verify ToArray() behavior when the stream's position is non-zero.
[Fact]
public void MemoryStream_ToArrayWithNonZeroPositionTest()
{
// Arrange
byte[] testData = new byte[] { 10, 20, 30, 40, 50 };
using ChunkedMemoryStream cms = new(this.allocator);
cms.Write(testData, 0, testData.Length);
long nonZeroPosition = 2;
cms.Position = nonZeroPosition;
// Act
byte[] actualArray = cms.ToArray();
// Assert that the returned array equals the full stream content
Assert.Equal(testData, actualArray);
// Assert that the stream position is still the non-zero offset
Assert.Equal(nonZeroPosition, cms.Position);
}Ensure that this new test is placed in an appropriate area within the file so that it runs with the other tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prerequisites
Description
Fixes SixLabors#2806
ChunkedMemoryStream contained multiple writing bugs and was too costly to fix/maintain relative to performance benefits so I'm just ditching it.
Greptile Summary
Complete rewrite of ChunkedMemoryStream to fix critical bugs and improve non-seekable stream handling in ImageSharp's encoding pipeline.
ChunkedMemoryStreamto use a simpler List-based chunk management system instead of linked listsImageEncoderto properly use chunked memory stream for non-seekable stream encodingawait usinginEncodeWithSeekableStreamAsyncNonSeekableStreamto correctly delegate write operations to underlying streamSummary by Sourcery
Completely rewrite ChunkedMemoryStream to simplify its implementation, fix bugs, and improve performance for non-seekable stream handling in ImageSharp's encoding pipeline.
New Features:
Bug Fixes:
Enhancements:
Tests:
Chores: