Shrink the hashfile: index file uniqueness by path hash - #2
Merged
Conversation
The files table enforced UNIQUE(filename), whose automatic index stores a second full copy of every path. Measured on a 20k-file tree that index was 1.75 MB - 24% of the whole hashfile - and it exists only to keep one row per path (for the INSERT OR REPLACE upsert, rename, and -R removal); nothing needs the path *text* for those. Store a 64-bit csum_path(filename) alongside the path and enforce UNIQUE(path_hash) instead. The path text is still kept (files must be opened by name to dedupe), but the uniqueness index now costs 8 bytes/row. The upsert, rename and delete-by-path queries key on the hash (delete keeps a filename tie-breaker so a hash collision can't remove the wrong row); a collision could at worst cause one unrelated file to be rescanned, never data loss. Bump the hashfile minor version so old files are rejected cleanly. Hashfile size (vacuumed): -11% short paths, -16% typical (~68 char), -28% long paths (~130 char); the UNIQUE index itself shrinks ~79%. Scan speed is unchanged. All unit and integration tests pass, plus new test_pathhash.py covering path-reuse eviction and -R removal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WhuZfCNECmgdb8FozGKagi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
filestable enforcedUNIQUE(filename), whose automatic index stores a second full copy of every path. On a 20k-file tree that index was 1.75 MB — 24% of the whole hashfile — and it exists only to keep one row per path (for theINSERT OR REPLACEupsert, rename, and-Rremoval). None of those need the path text.This stores a 64-bit
csum_path(filename)alongside the path and enforcesUNIQUE(path_hash)instead. The path text is still kept (files must be opened by name to dedupe), but the uniqueness index now costs 8 bytes/row. The delete-by-path query keeps a filename tie-breaker, so a hash collision can't remove the wrong row; at worst a collision causes one unrelated file to be rescanned, never data loss. The hashfile minor version is bumped so older files are rejected cleanly.Measurements (vacuumed hashfile)
The
UNIQUEindex itself shrinks ~79%. Scan speed is unchanged (within 1%, measured interleaved).Testing
All unit and integration tests pass, plus new
test_pathhash.pycovering path-reuse eviction and-Rremoval — these pass identically on this branch and on stock master, confirming behavior is preserved. The version guard rejects cross-version hashfiles cleanly in both directions.🤖 Generated with Claude Code