combine small files into composite blocks #151
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Give it its own loop for walking the tree, so that it can combine small files.
To iterate a local tree's subtree, start directly there and descend, rather than walking the whole thing and throwing data away. MergeTrees works on entry iterators not trees.
No longer uses WriteTree trait
Codecov Report
@@ Coverage Diff @@
## main #151 +/- ##
==========================================
+ Coverage 83.65% 84.37% +0.72%
==========================================
Files 35 35
Lines 3402 3482 +80
==========================================
+ Hits 2846 2938 +92
+ Misses 556 544 -12
Continue to review full report at Codecov.
|
This changes the backup file-count stats to include files that later failed to be stored.
All we need here is a reusable buffer.
The backup code needs to know about index hunk boundaries so that it can combine and parallelize storage of data within that hunk. And so the higher-level backup code now tells the index when it's done.
Previously the API required they be added to the IndexBuilder in order, but that gets in the way of grouping together small files and parallelizing compression. Instead, allow them to be added in arbitrary order within the hunk, and the IndexWriter will sort before serializing. Backup code is still responsible for keeping them in order across hunks, and the IndexWriter checks this. Also, rename IndexBuilder to IndexWriter.
Factor out code to read with retries
sourcefrog
force-pushed
the
combine
branch
2 times, most recently
from
05e020f
to
f8d8d96
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #66