-
Notifications
You must be signed in to change notification settings - Fork 55
De‐Duplication
This page explains how Downloader For Reddit's de-duplication feature works, what to expect from it, and what its limitations are.
When a file is downloaded, the app calculates an MD5 hash of its contents. This is a short, unique string based on the file’s exact binary data. The hash is stored in the app's database. Later, if a file with the same hash is encountered, the app treats it as a duplicate.
How duplicates are handled depends on your settings. You can choose to:
- Delete the duplicate file
- Move it to a separate folder
- Ignore it and keep everything
This feature is optional and can be turned on or off in the app settings.
If duplicates are not a concern for you, you can turn hashing off entirely. With hashing disabled, no files are hashed and no hashes are stored in the database. This will lead to slightly better download performance.
MD5 hashing creates a consistent fingerprint for a file based on its contents. If two files are exactly the same, their hashes will match. If they differ by even one byte, the hashes will be completely different.
This means that duplicate detection is based on exact file matches. The app does not compare images visually or check for near-duplicates, it only compares the underlying file data.
You might notice two files that appear identical but aren't flagged as duplicates. This is usually because of differences in the file data that don’t change how the image looks. For example:
- The same image saved as a PNG and a JPG will have different hashes.
- Two images with different compression levels or metadata (like timestamps) will also hash differently.
- Cropping, resizing, or even re-saving the same file with different software can change the hash.
As a result, visually similar files might not be treated as duplicates.
This approach is intentional. It keeps the process fast, lightweight, and predictable.
This feature works well for filtering out reposted content that’s been shared multiple times without any changes. It’s particularly helpful if you’re downloading a large volume of files from the same user who may have posted the same content to different subreddits.
You can configure the de-duplication behavior in the settings. Available options:
- Delete: Only one copy is kept. It will be the first one downloaded.
- Move: Duplicates are saved in a different folder. The directory path and the name of the duplicate can be set using the same tokens available for all downloads.
- Keep: All files are saved, regardless of their hash, in the download folder.
- Disabled: No duplicate detection is performed, no files are hashed, and no hashes are saved.
Choose the one that best fits your preference.
If you're unsure which setting to use, moving duplicates to a separate folder is a safe starting point. You can review them later and decide whether to keep or discard them.
Keep in mind that the database must remain intact for hash comparisons to work correctly. If you delete or reset the database, previously seen hashes will be forgotten.
If you run into a situation where files you expect to be duplicates are not detected, it's usually due to differences in file format, compression, or metadata.
The hashing will only take place for files downloaded after the v3.17.0 update. Previously downloaded files will not be hashed and you may download new content that is a duplicate of files downloaded before this update.