[Feature] Initially read and eventually write face metadata #4348
Replies: 15 comments 18 replies
|
This seems like it would be very difficult to implement and confusing. Camera manufactures include face recognition in their raws and every single software program handles face recognition tagging differently. The standard exif tags |
|
I would respectfully disagree. I feel that the MWG face regions are sufficiently standardised to be called a standard and they include the name of the person in the tag. Tons of tools (both free and paid) from various vendors (Including exiftool) support them. I believe face metadata should be transferrable from one tool to another and live with the photo much like we expect creation date and location to be transferrable. Obviously its ultimately the devs decision whether they want to support transportable metadata. |
|
@rhatguy I would also like that person metadata can be easily transferred, but my experience is that while Lightroom (for example) will write the metadata it will not read it properly. For that matter, in RAW files, camera vendors use different makernotes in order to identify facial tagging (unfortunately). I didn't realise the extent of the support for this metadata feature in other tools as I don't use much outside of Lightroom, Google Photos, and Immich. |
|
I am in alignment with rhatguy on this request. It would be very valuable. Also, as he noted, the MWG face regions are widely used - including in Lightroom and DigiKam, which are two of the largest tools in the market. |
|
Respecting standards should not even be a feature request |
|
I don't have any personal experience with Lightroom or different camera models specifically, but I feel like the "standards" for storing face metadata are fairly standardized at this point given the number of industry tools that have adopted them. When a tool like exiftool adopts something like this I fee like it lends at least some legitimacy to the format. Then other tools like digikam, Mylio, and PiGallery2 have adopted and share metadata between the tools. I agree Immich's face recognition good. Its honestly one of the better ones I've seen from online tools. That being said, its nowhere close to the accuracy of the curated tags I have embedded in my images. My dad for instance is tagged in 545 pictures, however I recently re-ran a full Immich scan (using the new antelopev2 model) and no only did I loose all my previous face tags in Immich (which I knew would happen, it just highlights the need to store the metadata for future use), but Immich only detected 426 pictures of my dad. Sure my workload to get those 426 tags was significantly less than the work I've put in creating the 545 MWG region tags, so I'm not minimizing the usefulness of Immich's face tagging. But, that doesn't fundamentally change my stance that I believe all metadata should be stored inside the image (or in associated XMP files). Otherwise, when Immich changes models again, or I move to another photo program/app, I loose all the metadata I've created and have to start over again. Storing the metadata with the image is a requirement for me to use any image tools at this point as tools come and go, but the pictures (and their metadata) are "forever". |
|
Here is the documentation by the Metadata Working Group (in archive.org bc their website is down): The relevant section for this discussion is "5.9. Image Region Metadata" (page 51). edit: I found another newer (2019) spec by the IPTC: |
|
I would also love this feature. I have a hundred thousands of photos with existing XMP tags. I meticulously worked for years on them using Picasa. This is the only thing stopping me from switching to Immich. |
|
I agree that face recognition is part of the metadata and should belong to the photo. Whether it is in a sidecar (like Immich already does with Coordinates) or in the file itself is probably a matter of taste. I would really like to see a solution to have the detected faces (which works absolutely fantastic for me in Immich by the way!) somewhere stored in or beside the files, so every other software I use with my huge library can benefit from it (and Immich could benefit with what was tagged over the last decades already by me). |
If is already in a sidecar why can't other software read it already? |
I did not say other software can not read xmp sidecars. Immich is writing coordinates to sidecars, but not the face / person information. The latter one is what people in this topic would like to have, if I did not understand something incorrectly. |
|
Facial data will need to have embedding information so that we can utilize the clustering mechanism to group them together when new faces are detected. AFAIK, sidecar don't have that info |
|
If i understand correctly, the embedding information is information about the detected face (its characteristics?) and probably very specific to how face detection is handled in Immich. I totally understand that it makes no sense to put that into an XMP. It would be nice, however, once a detected face gets a name tag in Immich, to have that name tag attached to the picture in some way. Other software that allows searching by tags would benefit from it and allowing the user ot search for a picture that includes a tag with a certain name. Sure, the embedding information is lost, probably also which face on the picture the tag belongs to, but it would retain a very important piece of information of a picture: This picture includes a person with name x and a person with name y, even when switching platforms. |
|
I would very much like to see this written into the image's xmp metadata like digikam and a lot of other software does (not a sidecar). The that usually contains the region and the person's name (+ I think email address?) This is one of the big blockers for me moving off Google photos. If I'm going to go through retagging all my photos again I want it to be portable and I already have a lot of old (pre Google photos) photos tagged in my library with xmp metadata from digikam and the old windows live photo gallery (they used compatible metadata). Otherwise you end up having to retag everytime software stops being supported or you want to move away. Ideally all metadata in the program should be copied into the photo file itself for portability |
|
@alextran1502 can you help us understand what "embedding" information is? Even if we didn't have embedding information for image metadata, wouldn't we just assume that the metadata from the picture is correct and display in the gui faces along with the names pulled from image metadata? I realize that would mean that a new face that was detected (by immich) could not be clustered against a face pulled from metadata, but we could still cluster multiple new faces and as soon as the user confirmed the actual name of that cluster couldn't the cluster of new faces be merged with the cluster of faces from existing metadata? |
|
An embedding is a list of 512 decimal values, usually between 0 and 1 I believe. It is heavily tied to the machine learning facial recognition model that's being used. We can pretty easily write person/face data to a sidecar file, but it is hard to incorporate both at the same time. If you did use metadata-only, you might be able to get it to work more easily, but merging the two together proves difficult. To begin with, the embedding column cannot be null otherwise the current solution faces all sorts of challenges. |
|
I would not store the embedding information in the xmp. That is application specific and can be saved in the app database.
The embedding can be generated from the tagged images for a new face when they are imported.
…________________________________
From: Alex ***@***.***>
Sent: Sunday, December 10, 2023 11:34:16 a.m.
To: immich-app/immich ***@***.***>
Cc: ericnotthered ***@***.***>; Comment ***@***.***>
Subject: Re: [immich-app/immich] [Feature] Initially read and eventually write face metadata (Discussion #4348)
Facial data will need to have embedding information so that we can utilize the clustering mechanism to group them together when new faces are detected. AFAIK, sidecar don't have that info
—
Reply to this email directly, view it on GitHub<#4348 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFIWTMQZ5T7W67DOWQW53RLYIYFDPAVCNFSM6AAAAAA5TQHPPKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMJTGEZDS>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Perhaps something like what Mylio does would make sense. When Mylio detects a face it uses a circle to denote the face. When a face is added manually it uses a square. Perhaps differentiating between a "Immich detected" face vs a face that was pulled from image metadata could use different shapes like this. Then if an "immich detected" face overlapped by ~90% with a metadata face you could assume the "immich detected" face was the same and just use the image metadata. In this case, I think the embedding information could stay inside the immich database and only standard MWG tags would be written to the image metadata. |
|
Does the photo metadata contain the face bounding boxes? Without this data, it is impossible to know which face is assigned to which person when the photo contains more than 1 face, which makes it incompatible with the current implementation of Immich machine learning. |
|
Yes, here is an example of how it looks both from the jpg file and from the xmp file. Note that I replaced the actual names with person1-4. These entries correspond with 4 different faces in this photo. Note how Mylio is tagging the description as "mylio_face". This doesn't affect other software reading the data from the images though. I'm happy to provide some test files if its helpful. c:\exiftool-12.41\exiftool.exe IMG_0746.jpg
<mwg-rs:Regions rdf:parseType="Resource"> |
|
To summarize: The MWG regions identify the bounding box of the face, and provide Name and Description tags.
In terms of workflow, I would suggest as a MVP:
Image import:
* Read image MWG regions from image or sidecar file XMP data.
* If name does not exist in db, create new.
* If exact region already exists in db (same name and bounding box), skip it.
* Add MWG regions to db for each applicable name.
* Calculate embedding for names with added regions.
* ++: On import/refresh, option to ignore MWG regions with names that match faces already identified in that image (ie, avoid creating duplicates).
* ++: On import/refresh option to ignore all MWG regions
* ++: On refresh, option to delete all Immich db regions for the image and create from file/sidecar MWG regions.
Face region added by Immich:
* Write MWG region to XMP data in file and/or sidecar.
Immich face detection:
* Option to ignore any regions with names that match faces already in the db for that image ie, avoid creating duplicates).
* ++: Option to delete existing face regions in image/sidecar file MWG data and write Immich regions.
Note that the requirements to avoid creating duplicates regions could be extended to allow defining a minimum region intersection between the regions in order for the new region to be ignored.
…________________________________
From: rhatguy ***@***.***>
Sent: Wednesday, December 13, 2023 11:44:35 AM
To: immich-app/immich ***@***.***>
Cc: ericnotthered ***@***.***>; Comment ***@***.***>
Subject: Re: [immich-app/immich] [Feature] Initially read and eventually write face metadata (Discussion #4348)
Yes, here is an example of how it looks both from the jpg file and from the xmp file. Note that I replaced the actual names with person1-4. These entries correspond with 4 different faces in this photo. Note how Mylio is tagging the description as "mylio_face". This doesn't affect other software reading the data from the images though. I'm happy to provide some test files if its helpful.
c:\exiftool-12.41\exiftool.exe IMG_0746.jpg
Region Applied To Dimensions W : 4000
Region Applied To Dimensions H : 3000
Region Applied To Dimensions Unit: pixel
Region Type : Face, Face, Face, Face
Region Name : Peron1, Person2, Person3, Person4
Region Description : mylio_face, mylio_face, mylio_face, mylio_face
Region Area X : 0.629789, 0.888162, 0.397451, 0.179784
Region Area Y : 0.349478, 0.316372, 0.342651, 0.365867
Region Area W : 0.062249, 0.061405, 0.073525, 0.079395
Region Area H : 0.107548, 0.110551, 0.121764, 0.190828
Region Area Unit : normalized, normalized, normalized, normalized
Person In Image : Person4, Person4, Person1, Person2
more IMG_0746.xmp
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:AppliedToDimensions
stDim:w="4000"
stDim:h="3000"
stDim:unit="pixel"/>
mwg-rs:RegionList
rdf:Seq
rdf:li
<rdf:Description
mwg-rs:Type="Face"
mwg-rs:Name="Person1"
mwg-rs:Description="mylio_face">
<mwg-rs:Area
stArea:x="0.629789"
stArea:y="0.349478"
stArea:w="0.062249"
stArea:h="0.107548"
stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>
rdf:li
<rdf:Description
mwg-rs:Type="Face"
mwg-rs:Name="Person2"
mwg-rs:Description="mylio_face">
<mwg-rs:Area
stArea:x="0.888162"
stArea:y="0.316372"
stArea:w="0.061405"
stArea:h="0.110551"
stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>
rdf:li
<rdf:Description
mwg-rs:Type="Face"
mwg-rs:Name="Person3"
mwg-rs:Description="mylio_face">
<mwg-rs:Area
stArea:x="0.397451"
stArea:y="0.342651"
stArea:w="0.073525"
stArea:h="0.121764"
stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>
rdf:li
<rdf:Description
mwg-rs:Type="Face"
mwg-rs:Name="Person4"
mwg-rs:Description="mylio_face">
<mwg-rs:Area
stArea:x="0.179784"
stArea:y="0.365867"
stArea:w="0.079395"
stArea:h="0.190828"
stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>
</rdf:Seq>
</mwg-rs:RegionList>
</mwg-rs:Regions>
Iptc4xmpExt:PersonInImage
rdf:Seq
rdf:liPerson4</rdf:li>
rdf:liPerson2</rdf:li>
rdf:liPerson1</rdf:li>
rdf:liPerson3</rdf:li>
</rdf:Seq>
</Iptc4xmpExt:PersonInImage>
</rdf:Description>
</rdf:RDF>
—
Reply to this email directly, view it on GitHub<#4348 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFIWTMW6DTMMJFQ6XZ4NHNTYJIASHAVCNFSM6AAAAAA5TQHPPKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQNBWGUZDQ>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
This looks perfect! |
|
Has there been any progress with this? I would love to move to Immich but having to redo all my face metadata is the only thing stopping me. |
|
Same would love to have all the data in xmp so I can reinstall easy immich because all the time consuming task are already in the files |
|
If it's useful here's the ticket from when digikam implemented it with some sample photos, etc |
|
Writing this data back to the file is the feature that would make me pay for an immich license. The process of retagging photos everytime you have to change software is always the blocker that makes it hard to change software and in trying to get away from google photos I want my photo files to be portable. At the moment I'm using digikam's face tagging and external libraries so I can use Immich as the "view" UI. But Immich's face detection and matching appears to be a lot better than Digikams and would avoid having to fire up a desktop program to tag. |
|
I don't think this would be an issue if there are writable XML files supported. They act more or less the same and tend to be very widely accepted and are basically a standard. I think it's better that originals are not written, but I guess it could be a use selectable flag if necessary. |
|
IMO separate sidecar files are much more susceptible to bit rot and getting out of sync with the photo file. Obviously a personal preference but I want my photo files to be portable. |
|
Any system without the necessary error correction is vulnerable to bitrot, if that's even a problem at all, but I see the point. I consider XML to be portable, but there is no reason both cannot be an option, with XML remaining a default. |
|
Reading this data is now supported, so I'll close this request and I've opened #15083 to track writing it. |
|
I also created #15107 for merging immich detected faces and metadata and their linked groups. |
Uh oh!
There was an error while loading. Please reload this page.
The feature
It seems like the original discussion around face metadata was closed so opening up a separate request here.
It would be great for Immich to be able to read existing face metadata from pictures (or XMP files). Examples of other software which read face metadata include PiGallery2 and Digikam. At worst case this data could be used as "hints" to Immich's face recognition system where if there was a significant overlap with and existing face region and an Immich discovered face region Immich might assume the existing metadata to be correct and suggest or even automatically tag that region in Immich's database. At best case Immich could use/display the actual face regions from either photos metadata or from accompanying XMP sidecar files.
Secondarily, it would be nice if Immich could write face metadata that was created/modified using Immich in a fashion which could be re-used by other photo management software. An example of an existing software that allows this is Mylio. Mylio reads existing face regions from JPG photos EXIF for instance when images are added, and creates its own XMP files. Then if faces are modified or created inside Mylio it writes those face regions to the XMP files. This way the original image is not modified. This seems to be a fairly acceptable way to ensure a program like Immich isn't responsible for destroying the image itself.
Finally the awesome exiftool is able to take face regions that are written to xmp and write them back into JPG files using a command similar to the below. This allows users who create face regions in XMP (with Mylio for instance) to easily merge those regions into the JPG images through a batch process if they choose.
exiftool.exe -r -ext jpg -overwrite_original -tagsfromfile %d%f.xmp -all:all -wm cg c:\test
Platform
All reactions