[Feature] duplicates check, default keep HEIC other than JPEG

[piscesvivian]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

For now, duplicate check keep lastest modify time (I guess) or largest file size by default.
as I know HEIC is More efficient than jpeg, HEIC image file size is about 50% less than JPEG , and some similar file were convert from HEIC to JEPG by icloud automatic，to get more device compatibility.

So, can we add a file type priority?
In most case HEIC is the best choice: Apple Original and smaller file sizes.
For thousands of photos, change default setting by click one by one it's too inefficient, it may take hours or days to deal with those photos.

Platform

• Server
• Web
• Mobile

[vw-kombi]

It’s been a year since I first posted about this when I started with Immich. This is still stopping me rolling this out. I can only assume either iPhones are not a priority, or this issue is not widespread but with my old method of adding photos since heic was introduced as default in iPhone, and I converted to jpeg on pc when copying them off, but left on iPhone as heic and now duped. By Immich. Heic are the default, and better on an iPhone so this should not be a feature but a default behaviour if you have an iPhone.

[stevenmcastano]

This isn't actually about immich itself... but I've been having the exact same problem and I got a bit tired of waiting! So... and let me preface this buy saying I'm not a developer... at all... but I've been giving it my best shot!

I've been working on messing around with an app called Immich Duplicate Finder which was created by @vale46n1 and I've been loving it so far... check it out, it's not exactly the same... no stacking option, and it only compares 2 images at a time... but I've been hacking away on the python code, one of the only languages I know decently well... and I've gotten it to the point there it will detect if there's an HEIC and it's extracted JPG and show you in the UI so you can better pick which one to toss.

No auto delete yet, not much in the way of config yet... but I'm getting there. If anyone would be willing to help... so show me how to contribute similar code to this project, I'd be happy to pitch in more... and if it's something you're interesting in testing, I'll do what I can to package up the pretty hacky code and get a fork posted!

Normally it highlight differences in a color with green being the "better" one. When it finds HEICs and JPGs and notices they are dups, it usually only highlights the size and makes the larger one green thinking its got more data so it must be better even though everything else matches.

This is the original output from the code in the current repo:

Now, with my new code you can see I added the "GPS Data" section to you can tell if you're delete a copy with GPS data when the other one doesn't have any, I also added the "Camera Info" sections which it puts together from EXIF data in the image, not just the "lens" tag... then the big win for me is that it now can tell if it's an HEIC pair, colors the HEIC file green, make the JPG file red and changes the size coloring to yellow if they're different so you can still see there's a difference, but you'd probably still want to go with the HEIC anyway. Here's what that looks like:

P.S. The "EXT_DIFF_HEIC" text can be turned off... that's just running in a debug mode so I can more easily find examples while testing!

[vw-kombi]

I'm not a dev in any way shape or form, so I cant help, but I can say I am also still waiting, not only that but I have to make a decision of deleting media from icloud or going to their expensive 2TB option. I immich caters more for android users ? how hard is it to have a tick box that says 'prefer HEIC'.

[stevenmcastano]

My code kind of does prefer HEIC now. It doesn't auto-delete anything, but I can look at making that happen.

However, when comparing duplicates it does check each detected pair to see if it's the combo of an HEIC and JPG, and when found, it highlights the HEIC as the one you'd want to keep.

Are you looking for something that will literally "one click" find all of those pairs and toss the JPG version?

[vw-kombi]

Yep, def one click to remove all jpg if there is an heic. I was looking at this on my oc files rather than in Immich and the issue is worse there as in some cases there are two heic and one jpg. This is because I also use iCloudPD to copy from iCloud to a backup on my pc. And something there is an extra one again as the wife may want a copy on her phone. So keep any heic and dump the other heic and jpg.

[kallioaleksi]

For me there's the additional issue of iCloud sync creating thousands of duplicates (I have a part of the photos exported from iCloud to a iPhotos library), would have to go through them manually since the automatic backup doesn't mark photos as synced, it checks the library for them and if I bulk delete all HEIC duplicates, it resyncs them each and every time.

[Johnoaa]

I would like to amend this request to work with any filetypes. I have photo sessions with raw files where Immich will always suggest keeping the raw file due to file size whereas 90 percent of the raws are not worthy of keeping and should be deleted. The rest should be stacked.

[2q2code]

All in all, configurability of the inbuilt duplicate finder feature would be an excellent and most desirable feature, however it only really affects the onboarding process. Once it's solved, it shouldn't happen again. However, it would be a useful tool if we were able to select the metrics upon which the deduplication decision was made. I would have thought that whether or not a photo was in an album would have been prioritised as well. In theory, provided the images are the same file type, larger is better to keep, but this case is more nuanced and other EXIF data may be able to assist in the decision.

[2q2code]

This may be of interest to those on this thread. Use with caution. https://gist.github.com/dahool/7cdd66d17b1b26c885c298f0c324d88c

[vw-kombi]

Have no idea what to do with that code. But if it’s only 64 lines long, maybe it’s easy for the Immich app guys to get in the base app ? It’s Over a year and I have not been able to onboard yet. I was hoping Immich-go would step in and cover this but not as yet I believe. I have wiped my config a few time and tried a few things on re importing but there are just too many dupes…. It’s all due to my workflows in the past. Added to that, as of 2 years back, we are using iCloud and icloudpd to backup locally, that’s another copy of the HEIC photos older than 2 years back too…..

Im thinking one day I may just have to re-sort all photos and fix the dupes on the source….. I worked at it briefly and it was a load of work.

[2q2code]

Tell you what - I intend to fiddle with this code to fix up my issue. If you aren't entirely unfamiliar with the linux terminal, I can send you some steps once I'm done (if successful) if you'd like?

[vw-kombi]

Cool. Yesterday I thought I had a temp solution - convert all my JPG's to HEIC on my PC source files....... Then delete and re-import all stuff. - But there is no batch offline converter for that......

[2q2code]

See my next comment - I spent a few hours making a more user-friendly version this morning. Hope it helps you out!

[2q2code]

Here's a slightly more user-friendly version of the script I linked to before. I hope it helps those stuck in this muddle. Installation instructions and guidance are in the comments at the top of the script and in the help for the script when run.

https://gist.github.com/2q2code/331265beed478d8d150781d8bc43536c

[stratacast]

I look forward to when this feature makes it. I have many gigs of family photos that are pre-immich that were saved to Nextcloud as jpg, but were never deleted from the phone. So there will be the on-phone HEIC (keep) and the Nextcloud jpg (trash). I know each user will have thousands of these to go through. So to just say deduplicate but keep the HEIC ones would be fantastic