Duplicate Tool
The Duplicate Tool finds duplicate and near-duplicate images in your dataset and helps you remove the extras while keeping the best copy of each.
Training on repeated images can bias a model towards whatever those images contain, so removing duplicates is often a useful step when preparing a dataset. The tool scans your images, groups the ones it considers duplicates, and lets you review each group before anything is deleted.
You choose how similar two images must be before they are treated as duplicates, which copy should be kept, and whether captions from the removed copies are merged into the one you keep.
The tool scans the current dataset. If you have opened a subfolder, the scan is limited to that folder. When a Gallery View search filter is active, the scan is further restricted to the files that filter matches.
Choose a detection method and your other options in the sidebar, then select Scan Folder to begin. As the scan runs, Studio reports its progress and streams in exact-match groups as they are found, so you can begin reviewing before the scan has finished.
A scan runs in the background. You can leave the tool and return to it later without losing the results, and a scan already in progress will be picked up again when you reopen the tool.
Studio offers four detection methods. Each makes a different trade-off between speed and how much editing it can see through, so the best choice depends on how your duplicates were created.
| Method | Speed | Best for |
|---|---|---|
| dHash | Fast | Identical or resized images. |
| pHash | Medium | Crops and compression, using a DCT-based perceptual hash. |
| Feature Match | Slow | Heavily edited or rotated images. |
| Content Hash | Fastest | Byte-for-byte identical files only. |
A difference hash comparing horizontal gradients across the image. It is quick to compute and reliable for images that are identical or have simply been resized.
A perceptual hash based on the discrete cosine transform (DCT). It is more robust than dHash to cropping and compression, at some cost in speed.
Compares local image features rather than a single hash. This is the most tolerant method and can match images that have been heavily edited or rotated, but it is also the slowest and is best reserved for smaller collections.
Uses the content fingerprint that Studio already computes for each file. Because these fingerprints are worked out in advance, this method is extremely fast, but it only matches files that are byte-for-byte identical. Two visually identical images saved in different formats or at different quality settings will not match.
Note: Because Content Hash is an exact match, the Likeness control does not apply to it and is disabled when it is selected.
The Likeness slider sets how similar two images must be, as a percentage, before they are grouped together as duplicates. It applies to the dHash, pHash, and Feature Match methods.
- At 100%, only exact matches are grouped.
- Lower values allow progressively looser matches, catching more near-duplicates at the risk of grouping images that are merely similar.
You can adjust Likeness after a scan has finished. Studio re-groups the existing results without re-scanning, so you can tune the threshold and watch the groups change without waiting for another full pass over your images.
When Studio groups a set of duplicates, it nominates one image as the keeper and marks the rest for deletion. The Keep Priority setting decides which image becomes the keeper.
- Largest Dimensions: Keeps the image with the highest resolution.
- Largest File Size: Keeps the largest file on disk.
- Oldest Modified: Keeps the image with the earliest modification time.
- Newest Modified: Keeps the image with the most recent modification time.
Like Likeness, this can be changed after a scan and the groups will re-sort without re-scanning.
When a duplicate is removed, any caption it carries would normally be lost. The Merge captions on delete option preserves that work by cleaning and merging each removed image's .txt sidecar into the keeper's caption before the duplicate is sent to the trash.
You can also set a Min words per line value. Cleaned caption lines with fewer words than this are dropped during the merge. Set it to 0 to disable the filter and keep every line, including short tag-style captions such as 1girl, blonde hair.
Once a scan completes, Studio shows one duplicate group at a time. The keeper appears on the left and the copy proposed for deletion appears on the right, each labelled with its filename and resolution.
Use Prev Group and Next Group to move through the groups. The counter shows your position, such as 3 / 47.
For the group you are viewing, you can:
- Delete This: Moves the shown duplicate to the trash and keeps the keeper.
- Not a Duplicate (Keep All): Dismisses the entire group without deleting anything. Use this when Studio has grouped images that you consider distinct.
When you are confident in the scan results, Delete All Duplicates removes every marked duplicate across all groups in one operation, keeping one image from each group. Studio asks you to confirm before it begins.
Note: Deleted duplicates are moved into the dataset's
.trash/folder rather than being erased. If a copy is removed by mistake, you can restore it from there.
If caption merging is enabled, captions from the removed copies are merged into their keepers as part of the same operation.