feat: add buffer recycling to checksum pipeline#3979
Merged
Conversation
Add a recycle channel from the main thread back to the I/O thread so consumed buffers are reused instead of allocating fresh ones on every read. This eliminates per-block heap allocations on the hot path. In pipelined mode, the I/O thread tries `try_recv` on the recycle channel before each read. If a buffer is available it is reused; otherwise a fresh buffer is allocated as fallback, avoiding any stall when the computation thread is slower than I/O. In synchronous mode, a single pre-allocated buffer is reclaimed from `current_block` on each call, cycling through idle -> in-use -> reclaimed without any allocation after the initial setup. Tests cover various file sizes (empty, sub-block, exact boundary, partial last block, many blocks) and property tests verify that pipelined and sequential checksums always match for random data.
oferchen
added a commit
that referenced
this pull request
May 18, 2026
* feat: add buffer recycling to double-buffered checksum pipeline Add a recycle channel from the main thread back to the I/O thread so consumed buffers are reused instead of allocating fresh ones on every read. This eliminates per-block heap allocations on the hot path. In pipelined mode, the I/O thread tries `try_recv` on the recycle channel before each read. If a buffer is available it is reused; otherwise a fresh buffer is allocated as fallback, avoiding any stall when the computation thread is slower than I/O. In synchronous mode, a single pre-allocated buffer is reclaimed from `current_block` on each call, cycling through idle -> in-use -> reclaimed without any allocation after the initial setup. Tests cover various file sizes (empty, sub-block, exact boundary, partial last block, many blocks) and property tests verify that pipelined and sequential checksums always match for random data. * style: remove unused RollingDigest import in property tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sender<Vec<u8>>) from the main thread back to the I/O thread so consumed buffers are reused instead of allocating fresh ones on every block readtry_recvon the recycle channel before each read - if a buffer is available it is reused, otherwise a fresh buffer is allocated as fallback to avoid stallingcurrent_blockon each call, cycling through idle/in-use/reclaimed without allocation after initial setupCloses #1759
Test plan
pipelined_matches_sequential_various_sizescovers 9 size variants including edge casesmany_blocks_pipelined_recyclingexercises 100 blocks of buffer recyclingsync_mode_reuses_bufferverifies synchronous buffer reusepipelined_equals_sequentialverifies equivalence for random data up to 512 KiBreader_collects_all_dataverifies no data loss for random data in both modes