A command-line tool that finds duplicate files in a directory tree and can reclaim the wasted space. It is plain Python with no third-party dependencies.
Comparing every file against every other file would be slow, so dedupe is careful about what it reads:
- Group files by size. Files of different sizes can never be identical, so most candidates are ruled out for free.
- For same-size groups, hash only the first 64 KB. That eliminates the majority of near-matches without reading whole files.
- For files that still look alike, compute a full SHA-256 to confirm they are identical byte for byte.
By default it only reports. Deleting takes an explicit --delete flag, and it always keeps one copy of every group.
# report duplicates under a folder
python3 dedupe.py ~/Downloads
# ignore anything smaller than 1 MB
python3 dedupe.py ~/Downloads --min-size 1048576
# machine-readable output
python3 dedupe.py ~/Downloads --json
# delete duplicates, keeping one of each (asks for confirmation)
python3 dedupe.py ~/Downloads --delete
# delete without the prompt
python3 dedupe.py ~/Downloads --delete --yes- Symlinks are skipped, so it never follows links out of the tree.
- Deletion keeps the first file in each group and only removes exact, hash-verified duplicates.
- Without
--delete, nothing is ever changed.
python3 -m unittestMIT