release chunk data memoryviews after use, see #1755 - #9978
Merged
ThomasWaldmann merged 2 commits intoJul 30, 2026
Conversation
On pypy, a memoryview over a C-API-created bytes object pins the buffer via cpyext: if the consumer drops the memoryview without calling .release(), the memory is never reclaimed - not even by gc.collect(). Since the chunkers wrap every yielded chunk in a memoryview, borg create leaked roughly 2-3x the processed data volume on pypy (a 20 GB create grew to a 43 GB memory footprint and got killed by the OS). Add release_chunk_data() to chunkers and call it at all places that consume chunker output: ChunkBuffer.flush, ChunksProcessor.process_file_chunks, ArchiveRecreater.chunk_processor, transfer --rechunk and the chunker benchmark. Explicitly releasing also frees the buffer timely on CPython instead of relying on refcounting of the last reference. Document the consumer obligation in the Chunk docstring. Memory now stays flat for large creates on pypy (~160 MB for a multi-GB create instead of linear growth). Also fix a pre-existing AttributeError in transfer --rechunk --dry-run: Chunk namedtuples have no .size attribute, all-zero chunks need chunk.meta["size"]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9978 +/- ##
==========================================
+ Coverage 85.83% 85.85% +0.02%
==========================================
Files 95 95
Lines 17034 17067 +33
Branches 2607 2611 +4
==========================================
+ Hits 14621 14653 +32
- Misses 1673 1674 +1
Partials 740 740 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Chunkers wrap every yielded chunk in a memoryview, but no consumer ever released it.
On pypy this is fatal: a memoryview over a C-API-created bytes object pins the buffer via cpyext, and if it is dropped without
.release(), the memory is never reclaimed — not even bygc.collect().borg createleaked ~2-3x the processed data volume (a 20 GB create reached a 43 GB footprint and was killed by the OS). On CPython, explicit release simply frees the buffer timely instead of relying on refcounting of the last reference.Changes:
release_chunk_data()helper inchunkers/reader.pyx; consumer obligation documented in theChunkdocstring.ChunkBuffer.flush,ChunksProcessor.process_file_chunks,ArchiveRecreater.chunk_processor,transfer --rechunk, chunker benchmark.AttributeErrorintransfer --rechunk --dry-runfor all-zero chunks (chunk.size→chunk.meta["size"]).Split out of #9976 (pypy support, see #1755) since it is useful independently of pypy; verified there by footprint sampling: multi-GB creates on pypy now stay flat at ~160 MB (was: linear growth to 43 GB for 20 GB input). Full test suites green on CPython 3.11 and pypy3.11.
🤖 Generated with Claude Code