perf: skip hashing files whose size no other file shares - #20
Merged
Conversation
Closes #12. scan() now runs in two passes instead of one. The first stats every file and groups by size; the second hashes only files that share a size with at least one other. Identical content implies identical size, so a unique size is proof that a file cannot be a duplicate, and reading it is provably wasted work. Grouping is still decided solely by the full SHA-256, so results are unchanged. Verified by diffing complete report output between the old and new binaries on a fixture covering duplicates, same-size-different-content, unique sizes, empty files and named copies: byte-identical. Measured on a 456-file / 425 MB fixture mirroring a real Desktop (98% of files having a size no other file shares, which is typical of screenshots): files hashed 456 -> 6 bytes read 425 MB -> 1.8 MB (99.6% less I/O) wall time 0.22s -> 0.00s Why this matters beyond CPU On macOS with iCloud Drive's "Desktop & Documents" and Optimize Mac Storage, files can be dataless placeholders whose contents live in iCloud. Opening one forces a download. A real Desktop measured 272 of 456 files dataless, 200 MB that the old scan would silently pull over the network - a stat-only pass took 0.115s where the full scan had not finished after 120 seconds. os.Stat does not materialise a placeholder, so the size filter avoids nearly all of that. The residual case, a size collision between two placeholders, is tracked as #19. Changes - scan.go: two-pass scanWithProgress; new candidate type; Report gains FilesHashed and BytesHashed. os.Stat is used for sizing (it follows symlinks) so sizes match what the single-pass scan saw. - Hash candidates are sorted by path before hashing, so progress and warning order stay deterministic. - report.go: files_hashed and bytes_hashed added to the JSON summary. - main.go: progress is now a single line rewritten in place ("Hashing 6/456") rather than one line per file. The old form emitted thousands of lines and rendering them dominated the run - measured at 0.227s vs 0.152s through a pty for 5000 files. Progress is suppressed entirely when stderr is not a terminal, where carriage returns would just be noise in a log. Behaviour change worth noting An unreadable file with a unique size no longer produces a warning, because nothing ever opens it. TestScanRecordsWarningForUnreadableFile was updated to give the unreadable file a same-sized sibling so it still reaches the hash pass, and TestScanDoesNotWarnAboutUnreadableFileWithUniqueSize was added to pin the new behaviour deliberately rather than by omission. Tests 56 tests (up from 51), race-clean, gofmt and vet clean. New tests assert that unique sizes are never hashed, that size collisions are hashed but not grouped, that real duplicates are still found with exact FilesHashed/BytesHashed counts, and that progress fires only for hashed files with a correct running total.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12. Turns the scan into two passes: stat everything, then hash only files that share a size with another. A unique size is proof a file cannot be a duplicate.
Measured
456-file / 425 MB fixture mirroring a real Desktop (98% unique sizes — typical of screenshots):
Why it matters beyond CPU
On macOS with iCloud "Desktop & Documents" + Optimize Mac Storage, files can be dataless placeholders whose bytes live in iCloud — opening one forces a download. A real Desktop measured 272/456 files dataless, 200 MB the old scan would silently pull over the network: a stat-only pass took 0.115s where the full scan hadn't finished after 120 seconds.
os.Statdoes not materialise a placeholder, so the size filter avoids nearly all of it. The residual case (two placeholders colliding on size) is #19.Correctness
Grouping is still decided solely by the full SHA-256 — the size pass is a pure filter. Verified by diffing complete report output from the old and new binaries over a fixture with duplicates, same-size-different-content, unique sizes, empty files and named copies: byte-identical. The existing golden tests also pass unchanged.
Progress rendering
Also replaced the per-file
Processing:lines with a single in-placeHashing 6/456counter — the old form emitted thousands of lines and rendering them dominated the run (0.227s vs 0.152s through a pty for 5000 files). Suppressed entirely when stderr isn't a terminal.Behaviour change
An unreadable file with a unique size no longer warns, because nothing opens it. Both the old and new behaviours are now pinned by explicit tests rather than left implicit.
56 tests, race-clean.
🤖 Generated with Claude Code