file_scan: grow hash arrays geometrically, not one element at a time - #95
Merged
Conversation
allocate_hashes() preallocates the block- and extent-hash arrays from a FIEMAP estimate. When that estimate fell short, add_block_hash()/store_extent() each grew the array by exactly one element (blocks_count++ / extents_count++, realloc), so a run of short estimates cost one realloc per element — O(n) reallocs and O(n^2) copying on a fragmented file. Factor the shared "ensure room for index+1" logic into ensure_hash_capacity() and have it double the array instead, so any estimate shortfall costs O(log n) reallocs. This also removes the pressure for the prealloc estimate to be exact (it's now a best-effort upper bound), and dedups the identical grow-by-one blocks in both add_block_hash() and store_extent(). No behavioural change to hashing or dedupe; verify.sh green (90 tests + valgrind smoke). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #94, addressing the deeper issue behind that PR's careful prealloc estimate.
What
allocate_hashes()preallocates the block- and extent-hash arrays from aFIEMAP estimate. When the estimate fell short,
add_block_hash()andstore_extent()each grew the array by exactly one element (count++then
realloc). A run of short estimates therefore cost onereallocperelement — O(n) reallocs and O(n²) copying on a fragmented file.
This factors the shared "ensure room for
index + 1" logic intoensure_hash_capacity(), which doubles the array instead. Any estimateshortfall now costs O(log n) reallocs. Both
add_block_hash()andstore_extent()had the identical grow-by-one block; they now share the helper.Why now
#94 rounded the prealloc estimate outward specifically so it would "never
under-count (which would trip the one-at-a-time realloc growth)". Geometric
growth removes that pressure: the estimate is now just a best-effort upper
bound, and a rare shortfall is cheap regardless. (#94's tighter estimate is
still worthwhile — it keeps growth from triggering at all in the common case —
so the two changes complement each other.)
Not a bug fix
No hash or dedupe result changes; only the allocation strategy for a
short estimate.
blocks_index/extents_index(the counts actually written tothe hashfile) are untouched.
Testing
scripts/verify.shgreen: clean build (warnings are failures), 90 tests pass,valgrind scan+dedupe+replay smoke clean.
🤖 Generated with Claude Code