People: Improve facial recognition for young children

[ScuttleSE]

The model used to categorize faces seems to be poorly trained with younger children.

I have a cluster with hundreds of faces ages 0-10, dozens of different people, but Photoprism thinks it's the same face.

[lastzero]

Yes, that's the case. Funding will fix this.

[lastzero]

Maybe we should exclude children from recognition, so that you need to name every face individually.

[lastzero]

Overall, we're happy to have such specific issues 🥳

[tootbrute]

Maybe we should exclude children from recognition, so that you need to name every face individually.

Another idea is make it easier to "undo" a bad match. I have a folder for a child's face. I clicked to see what photos it found of the face. 90% of the photos are the wrong person.

It would be great if I could select all the wrong photos, then go to the action button at the bottom and "clear faces".

Of course, it would be great if it recognized children better, but this might help people clean up their faces database in other instances as well. For myself at this moment, I think I may have to go back and just delete this person to remove all those faces that are wrong.

[lastzero]

We've added a special rule for baby faces to the latest update that excludes them from automatic matching / clustering. This won't change already indexed / matched faces though.

[tootbrute]

We've added a special rule for baby faces to the latest update that excludes them from automatic matching / clustering. This won't change already indexed / matched faces though.

Wonderful!

Now, what is the current suggestion on how to clear bad matches or delete a whole person? --> docker-compose exec photoprism photoprism faces index?

I hope I can avoid reindexing everything because with 50,000 pictures, that face matching hammers my server for 24 hours, but I can't think of another way.

I see I can hide a person, but not 'delete' that person. The only way I've found this is to go into each picture and click the 'eject' button on the person. This is not feasible with 100+ pictures.

[lastzero]

Now, what is the current suggestion on how to clear bad matches or delete a whole person? --> docker-compose exec photoprism photoprism faces index?

If you didn't invest too much time in naming faces, you may reset existing faces & people and then ONLY reindex faces:

docker-compose exec photoprism photoprism faces reset -f
docker-compose exec photoprism photoprism faces index

Skip the docker-compose exec photoprism prefix when not using Docker Compose, or you're running the commands from inside the container. You can't perform these operations from the UI.

I hope I can avoid reindexing everything because with 50,000 pictures, that face matching hammers my server for 24 hours, but I can't think of another way.

Is that the time for a full rescan or just the faces as described above? Reindexing faces takes ~6h for my 72k pictures (106k files).

I see I can hide a person, but not 'delete' that person. The only way I've found this is to go into each picture and click the 'eject' button on the person. This is not feasible with 100+ pictures.

Permanently deleting a face cluster isn't implemented in the UI - in particular because it can't be reversed, so it would be for advanced users only unless we manage to develop a really good UI explaining all the differences and options. Not quite there yet. Might come in a later release. For now, you can reindex faces only as described above (which will skip blacklisted clusters from the start) or hide unwanted faces / people until we have "permanent delete" implemented in the UI.

[tootbrute]

Thanks for the suggestions.

That was for just a faces scan. It maybe wasn't 24 hours, but definitely over 12 hours. I started it at 7 pm and it was still going at 7 am in the morning when I left for work. I think it was still running when I got home around 4 p.m. that day.

My server is not too new. It's a 2015 Lenovo Ideacenter with 8 GB RAM, but also running other self-hosted services on the box (Nextcloud, XMPP. WriteFreely, Searx, Wireguard, 2 instances of Photoprism), and my originals folder is on a RAID.

Maybe also I have lots of faces? I have lots of photos from school of random kids because I'm a teacher.

$ neofetch 
       _,met$$$$$gg.          
    ,g$$$$$$$$$$$$$$$P.       ---------------- 
  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux 10 (buster) x86_64 
 ,$$P'              `$$$.     Host: 10115 IdeaCenter Q190 
',$$P       ,ggs.     `$$b:   Kernel: 4.19.0-17-amd64 
`d$$'     ,$P"'   .    $$$    Uptime: 33 days, 9 hours, 21 mins 
 $$P      d$'     ,    $$P    Packages: 1257 (dpkg) 
 $$:      $$.   -    ,d$$'    Shell: bash 5.0.3 
 $$;      Y$b._   _,d$P'      Terminal: /dev/pts/0 
 Y$$.    `.`"Y$$$$P"'         CPU: Intel i3-2365M (4) @ 1.400GHz 
 `$$b      "-.__              GPU: Intel 2nd Generation Core Processor Family 
  `Y$$                        Memory: 2419MiB / 7878MiB 
   `Y$$.
     `$$b.                                            
       `Y$$b.
          `"Y$b._
              `"""

I'm curious what your test box is running?

[tribut]

I see I can hide a person, but not 'delete' that person. The only way I've found this is to go into each picture and click the 'eject' button on the person. This is not feasible with 100+ pictures.

I've run into the same problem. After "ejecting" some (5-ish) of the wrongly recognized pictures, the others disappeared as well. So if you want to keep your faces, maybe thats an option for you.

[lastzero]

I see I can hide a person, but not 'delete' that person. The only way I've found this is to go into each picture and click the 'eject' button on the person. This is not feasible with 100+ pictures.

I've run into the same problem. After "ejecting" some (5-ish) of the wrongly recognized pictures, the others disappeared as well. So if you want to keep your faces, maybe thats an option for you.

Indeed, clicking the eject button will report a conflict that will reduce the number of matches until none are left in theory. However, that's quite an expensive operation as many faces are going to be compared every time you click the button. Keep that in mind.

[lastzero]

I'm curious what your test box is running?

OS: Ubuntu 20.04.3 LTS x86_64 
Kernel: 5.4.0-88-generic 
Shell: bash 5.0.17 
CPU: Intel i3-4130 (4) @ 3.400GHz 
GPU: Intel 4th Generation Core Processor Family 
Memory: 7064MiB / 15698MiB 

Guess the clock speed makes a major difference. CPU is from 2013, about the same age as yours. Added an SSD for the database and caching, originals stored on traditional magnetic disks (RAID 1).

[acranox]

We've added a special rule for baby faces to the latest update that excludes them from automatic matching / clustering. This won't change already indexed / matched faces though.

Any chance we could get a toggle to enable/disable this? Maybe I'm misunderstanding whatever the change was, but the automatic matching was working well enough for me with kids, and it does seem that a more recent release isn't matching the kids as well as it used to. I see there's some discussion in #1627 as well. I don't need a tunable for every option, but I think there's some of us who will take the auto-clustering even knowing it isn't as accurate for kids as for adults.

[lastzero]

Yes, we could also make the comparison stricter (reduce the radius) so that less faces match to these embeddings:
https://github.com/photoprism/photoprism/blob/develop/internal/face/children.go#L4

I've added them because the results for those were unusable on my personal instance. Totally random baby faces all matched with these embeddings. Adding yet another config option is the last option for us. The already available options are confusing enough to most users and I'd need to write a book to fully document them.

The actual issue with training models and now implementing a workaround is missing test data aka baby face examples. It's really hard for us to do the right thing when working mostly blind. It probably takes a few updates until most users are happy with their personal photos (which we will never see).

Adding more options shifts the work to each individual user, who must then experiment on their own.

[rickysarraf]

Adding more options shifts the work to each individual user, who must then experiment on their own.

One config may not cover all use cases, as you already saw in this case. In my opinion, you should vouch for making it extensible (with multiple settings), while keeping a sane set of defaults. And also have a photoprism dump-config to list the effective config settings. Something that you should also then add to your bug template here.

[lastzero]

The model either has a blind spot there or not. It's not something you can configure away, that's why this issue exists. I suspect the radius was chosen too large by me as I didn't have enough test data to come up with better values. We wanted to do the creator of this issue a favor, and everyone else who complained.

Now we received feedback that it was too much (thank you for this!), which wasn't clear at that time. Most users wouldn't understand what an euclidean distance and embeddings are. They also wouldn't know what the impact is if they "just disable everything". We want this to work out of the box with the best possible result.

I'm currently working on multi-user support, which many sponsors are waiting for - otherwise you would have an improved version already. We do our best. Even interrupted dinner so that you get a quick reply! 😉

To display config options and their effective values:

docker-compose exec photoprism photoprism config

see https://docs.photoprism.org/getting-started/docker-compose/#examples

[lastzero]

Changed the matching radius in our Development Preview. Will be part of the next update.

[lastzero]

@rickysarraf Does photoprism config work for you?

[rickysarraf]

@rickysarraf Does photoprism config work for you?

Yes, Indeed. That's exactly what I thought of. Thank you.

[rickysarraf]

Changed the matching radius in our Development Preview. Will be part of the next update.

Thanks. I've spawned a separate test build now. Will test it over the next week.

[guma44]

Hi, is there a way to include children in clustering or this feature is disabled for good?

[lastzero]

Did you test our latest Development Preview?

[guma44]

Thanks! I tested it and I do not see big improvement. Eg. For me the problem is that eg. I have 10 photos that are the same scene (just taken multiple time) and because of the children setting (most probably) I have to label each of these photos as all the (same) faces are recognised as different. I would appreciate this parameter is tunable for more advanced users.

Edit:

Just to say something positive as the topic is about specific problem, I have to say that this feature (face recognition) is anyway awesome and it saved me hours of work :D.

[lastzero]

Before we add yet another config parameter (check how many we have already), it would be awesome to debug your issue using example files. We won't share them and delete them when done. Our email is hello@photoprism.app. Thank you!

[lastzero]

Since we didn't receive any test files so far, we continue with try & error. Radius has been further reduced. If the issue still persists it might be something else or you need more aggressive matching - for example by reducing the number of cluster cores from 4 to 3 (so that only 3 very similar faces are needed to form a cluster).

[guma44]

Hi, I prepared three sets of files. Let me send them today.

Edit:
Done. Good luck :D.

[ministryofsillywalks]

Is there any way to "batch add" a person to pictures that aren't stacked?
I have a lot of pictures of my kids and going into each one separately and adding the person information would take a long time ;)
It would be great for portraits where there is clearly just one person in the picture. Just select all of them and say "this is person xyz"

[lastzero]

see #1679 and #271

Not exactly this, but I'd say it could later be an option in batch edit. Right now, you could only use the API for this, but then you don't see the photos unless you also develop a custom UI (so lots of work).

[ministryofsillywalks]

Thanks for the info.
Maybe a an improvement that could be implemented:
When clicking on "Show all new faces" under the people tab it brings up all faces that can't automatically be tagged. This is great. However it does this in the standard "browse" view. Why not just set this to the "people view" to just show the faces like on the "new people view" on the page you just came from. Then you could easily add the correct names without having to go on the edit button of the image each time?

Extra Info and probably out of scope for the time being:
Lightroom does it like this. It shows all new unstacked faces that couldn't be matched and then you can directly add the name under each face. Here you can also easily select multiple identical faces and batch edit them. (which would also make it easier to batch edit images with multiple faces in them)

(This is what it looks like in LR Classic)

[graciousgrey]

Both is planned: #1679 and #1601 :)

[lastzero]

@rickysarraf Added an info box to the Config Options docs explaining how recognition works incl links to related research papers: https://docs.photoprism.org/getting-started/config-options/#facial-recognition

[Maltavius]

Siblings is another issue, my sons at age 2 looks quite much alike.
But most face recognition software wants to tag my oldest son on pictures taken 5 years after his birth after having older pictures already tagged.
What size of images do you want and how many for testing?

[layanto]

I too have the same issue as @Maltavius where my daughters have similar facial features at the same age.

Is it possible to include DOBs for subjects and face recognition algorithm uses this information (especially for young children)? For example, face embeddings for subject A at age 5 years old will match photos that are around 4-6 years from the subject's DOB. But will not match for photos where subject A would have been 8 years old so that they can match subject B at age around 5 years old.

The time tolerance should be correlated to subject's age. If the face embeddings is for baby or young child subject, it will only match with markers of photos taken +/- x months. While for adult subject, the time tolerance can be +/- x years.

[lastzero]

Last commit adds two unofficial env variables so advanced users can experiment:

1. PHOTOPRISM_FACE_KIDS_DIST=0.6950 (valid range: 0.1-1.5, -1 to disable)
2. PHOTOPRISM_FACE_IGNORED_DIST=0.86 (valid range: 0.1-1.5, -1 to disable)

Edit: The second variable has been changed from IGNORE to IGNORED. Please note that this is completely experimental and we cannot provide support. For advanced users only.

[IeuanK]

Would it be feasible to layer a per-year face cluster/model on top of the current one? For example, have a cluster for 'Peter Smith', but then if Peter was born in 2014, have one model for 2014, one for 2015, one for 2016 etc up until it matches the general model for that person, but all are linked to the same name?

I very much understand just turning off the baby face matching, but I have thousands of pictures of my children when they were very young. On top of this, it seems to get confused with children who are now adults as well. A way to structure a generational model could potentially fix this, but is it possible?

As for funding, is this a matter of buying a dataset, or does the model need to be trained specifically on baby pictures? Any way we can contribute training data without exposing the source pictures?

[lucasHabkost]

PHOTOPRISM_FACE_KIDS_DIST=0.6950 (valid range: 0.1-1.5, -1 to disable)

I'm passing that as an ENV parameter, but since I'm not and advanced user I left "0.6950" (I haven't experimented with any other number).

Do you think this has an impact on the situation of pictures almost identical and the child is recognized as 2 different people?
The adult there is recognized as the same person on both pictures.

[IeuanK]

PHOTOPRISM_FACE_KIDS_DIST=0.6950 (valid range: 0.1-1.5, -1 to disable)

I'm passing that as an ENV parameter, but since I'm not and advanced user I left "0.6950" (I haven't experimented with any other number).

Do you think this has an impact on the situation of pictures almost identical and the child is recognized as 2 different people? The adult there is recognized as the same person on both pictures.

I'm running into this same exact issue a lot, too. Pictures of my kids where sometimes it picks up one of them, but almost never all of them, even though they're the exact same picture (but different hash because different folder). Really seems like it should be able to at the very least do a 1:1 match like that?

[fjw]

Same issue here. Everything is well with adults but it doesn't detect my daughter at all.
I have hundreds of unrecognized images of her and I already tagged hundreds images of her but it still can't recognize her face on other pictures.

I donated additional 5$ to this, if this helps. But for some reason it doesn't show in the comments. Perhaps I did something wrong. I will check again in some days. I would try to contribute and help solving it myself, sadly I have no Go and ML experience...

[lastzero]

Note we've stopped using IssueHunt. You're welcome to support us on GitHub Sponsors and Patreon! 🤗

[LudwikO]

The actual issue with training models and now implementing a workaround is missing test data aka baby face examples. It's really hard for us to do the right thing when working mostly blind. It probably takes a few updates until most users are happy with their personal photos (which we will never see).

Can you tell more about it? How many photos are needed? Do they need to be tagged? Do you need many photos for a few kids or rather a few photos for each of a hundreds of kids? Can you please advise?

[lastzero]

See #3124 (comment)

Their "lite" archive contains about 60,000 faces to give you an idea. Ideally, you should tag them or organize them according to what you want the model to learn.

[lastzero]

Note that all issue subscribers receive an email notification from GitHub whenever a new comment is added, so these should only be used for sharing important information and not for discussions, questions or expressing personal opinions. Thank you! ❤️