Face training is ineffective

[docwisdom]

🐛 Bug Report

• 📁 I've Included a ZIP file containing my librephotos log files
• ❌ I have looked for similar issues (including closed ones)
• 🎬 (If applicable) I've provided pictures or links to videos that clearly demonstrate the issue

📝 Description of issue:

I have done considerable work "training" the face recognition model by associating hundreds of photos with dozens of people. When I run the train faces function, it does not infer any new photos for my people even though I have thousands more photos of those people in my library.

🔁 How can we reproduce it:

Associate a set of photos to a person, then click Train in the library drop down menu

Please provide additional information:

• 💻 Operating system: Unraid
• ⚙ Architecture (x86 or ARM): x86
• 🔢 Librephotos version: I dont know where to find this
• 📸 Librephotos installation method (Docker, Kubernetes, .deb, etc.): Docker/Unraid https://hub.docker.com/r/reallibrephotos/singleton
  • 🐋 If Docker or Kubernets, provide docker-compose image tag:
• 📁 How is you picture library mounted (Local file system (Type), NFS, SMB, etc.): local
• ☁ If you are virtualizing librephotos, Virtualization platform (Proxmox, Xen, HyperV, etc.):
  logs.zip

[scepterus]

Can confirm, after the first big batch, it rarely finds new matches for faces. It could be a size of the pool it's trained on for a specific face that exceeds the limits, or it could be the general limit to the size of the training data it can go through to match.

Haven't looked at the logic in the code, but since I also run docker, that's what I think the cause is. Also, if it is, maybe instead of having one major pool of face data, we can specify a pool for each face and have it iterate those pools when training.

[Rihcus]

Can confirm, after the first big batch, it rarely finds new matches for faces. It could be a size of the pool it's trained on for a specific face that exceeds the limits, or it could be the general limit to the size of the training data it can go through to match.

Haven't looked at the logic in the code, but since I also run docker, that's what I think the cause is. Also, if it is, maybe instead of having one major pool of face data, we can specify a pool for each face and have it iterate those pools when training.

By any chance did you run into a bug where new faces are tagged as unkown? I tried training/tagging 200 faces but now new similar faces are getting tagged as unkown (faces are in clear lighting conditions, looking at camera).

Also not sure if this is usefull but:

On my first scan I used hog
Then second/3rd scan used cnn

• midway 3rd scan I changed it from cnn back to hog (taking to long)

I also ended up with a rouge worker problem (ended up fixing by docker compose down setting the volume folder to a blank folder and letting the stuck process quite, then again docker compose down change env volume back to origional).

After this quick fix I ended up with the face scan getting stuck (library scan worked)

Doing a full rescan (via web ui) seemed to fix the face scan stuck issue but I noticed the inference system (using hog) isn't working.

Not sure if this overall is similar to your issue just curious if any similarities are present.

[scepterus]

I recently found out about cnn, and that has done a better job (with some fine-tuning) at finding faces, but it still hits a wall. No matter how many photos a person has, at some point it stops training on those photos and finding new ones that match.

[Rihcus]

I recently found out about cnn, and that has done a better job (with some fine-tuning) at finding faces, but it still hits a wall. No matter how many photos a person has, at some point it stops training on those photos and finding new ones that match.

I wonder if this convergence warning (1000) limit is related.

I switched back to hog but I'm not sure if the act of switching between hog and cnn corrupted the training data.

[scepterus]

As I learned in another issue, that might give you duplicate faces if you switch between them. as for that limit, looks like it, we need @derneuere to tell us for sure though.

[derneuere]

Face Detection, Face Recognition, Face Classification and Face Clustering are all different things.

1. Face Detection finds faces in pictures. Here you can choose between HOG and CNN. It does not change any following steps. It looks like @Rihcus changed the file permission of his thumbnails, which is why the system can't create preview pictures for the faces and fails. Please open a separate issue for that with steps to reproduce.

2. Face Recognition creates a 512 dimension embedding for a given face. It makes them comparable with each other. Right now we only have the one method and should be good enough.

3. Face Clustering does not take into account any previously attained knowledge, like already labelled persons. We switched to this system because we do not need any supervision from the user, leading to a better out-of-the-box experience. Clustering only changes, when you either delete / add faces or if you change the parameters in the setting. This is the original issue, he expects that it takes labelled persons into account, which it does not.

4. Face Classification is how we get the confidence values. This system learns from labelled persons, but currently only does do the confidence values. In the future we will add a mode, where it will match to only persons and not clusters, which is what you guys seem to expect. There seems to be in @Rihcus system an issue, where he has too many clusters. Without any additional information, I can't reproduce that.

Tldr: Play around with the settings for clustering and delete bad faces when you train.