|
Computing SHA-256 on gigabyte-sized files can be slow. How does DupeSweep filter out non-duplicates quickly? |
Answered by
amasen02
Sep 12, 2026
Replies: 1 comment
|
DupeSweep uses a 3-stage funnel: 1) exact file byte size matching, 2) 4KB head/tail sparse hashing using xxHash64 (processing GBs/sec), and 3) full cryptographic SHA-256 only on files that match stages 1 and 2. |
0 replies
Answer selected by
amasen02
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
DupeSweep uses a 3-stage funnel: 1) exact file byte size matching, 2) 4KB head/tail sparse hashing using xxHash64 (processing GBs/sec), and 3) full cryptographic SHA-256 only on files that match stages 1 and 2.