People: Facial Recognition

[lastzero]

As a user, I want to tag familiar human faces so that I can find all photos of these persons.

PhotoPrism must be able to detect and recognize faces. There are existing libraries for this, so we should try those first. See Face Recognition in our Wiki.

Here is a complete howto: https://hackernoon.com/face-recognition-with-go-676a555b8a7e

Acceptance Criteria:

• You SHOULD use go-face (unless you find a better library)
• A unit test demonstrating face recognition with real images and faces MUST be delivered
• A user interface MAY be developed and gives extra karma
• You MUST update our Docker images in /docker as needed (go-face requires dlib as external dependency); let us know when we shall push new images to Docker Hub
• Documentation MUST be updated so that other developers know everything to continue working on this task

---

IssueHunt Summary

lastzero has been rewarded.

Backers (Total: $900.00)

• issuehunt ($40.00)
• totempaaltj ($30.00)
• uluvil3gcs ($22.00)
• urbiops ($20.00)
• luxumbra27 ($10.00)
• cloudfreeshop ($50.00)
• kreusen ($10.00)
• setime ($4.00)
• spoorun ($100.00)
• friesoft ($10.00)
• conrad82 ($10.00)
• angelnu ($20.00)
• bitbacchus ($5.00)
• mauriceoegerli ($50.00)
• csev1755 ($5.00)
• ayancey ($5.00)
• ark- ($20.00)
• mastert ($5.00)
• $484.00 have been anonymously funded.

Submitted pull Requests

• #1541 People: Prepare for release #22

---

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[husayt]

Here are some more libraries to consider:

• Openface

• https://github.com/ageitgey/face_recognition

The latter is using Dlib state-of-the-art face recognition built with deep learning. The model has an accuracy of 99.38% on the Labeled Faces in the Wild benchmark

[lastzero]

I'd prefer a library implemented in Go to improve performance and reduce complexity.

[devzsolt]

As a proof of concept you can create a small interface to an external tool (eg. https://github.com/ageitgey/face_recognition using its CLI) and then later it's possible to replace it with a native Go lib. I think it would speed up the early development, this Python lib looks pretty much mature.

[IssueHuntBot]

@issuehuntfest has funded $40.00 to this issue. See it on IssueHunt

[xeoncross]

The easiest way to do this is simply to slap https://github.com/esimov/pigo into the project as a pure Go include.

[lastzero]

@xeoncross Thanks! But Pigo is just face detection, not face recognition...

See https://github.com/photoprism/photoprism/wiki/Face-Recognition

[lastzero]

This looks good to me on first sight: https://github.com/jdeng/goface

[trosel]

Could object detection be included in this request as well? Some libraries work with both faces and objects I think.

[lastzero]

@trosel We already have object detection / classification, without position however. Facial recognition is different from classification in that you need parameters for face parts as output to do the actual recognition based on a clusters (and not just detection).

What specific library do you refer to?

[cgudea]

I don't have any go experience but I'd really like to see this feature implemented. Is there anyone currently doing work on this? If so, would adding money onto the bounty help expedite the development?

[lastzero]

@cgudea You're most welcome to support our work via GitHub Sponsors:

https://github.com/sponsors/lastzero

No contributions there yet as GitHub Sponsors is pretty new. They will match any donation in the first year!

Don't expect to see a pull request for facial recognition anytime soon, it's too complex. We'll start working on it once albums and sharing are done. Financial support makes a major difference as we otherwise have to stop development frequently to do commercial projects.

[cgudea]

@lastzero totally understandable. I'm a huge fan of the goals of this project and will be looking to contribute after the holidays in January either financially or via development (I am a mobile developer at my dayjob).

[lastzero]

@cgudea Thank you <3

[ghost]

@lastzero If you want to get rid of the dependency of dlib, you can try the way that ownphotos does facial clustering. https://github.com/hooram/ownphotos/blob/dev/api/face_clustering.py

You can use any tensorflow CNN model that extracts facial features, and do a kmeans on them with https://github.com/muesli/kmeans

[lastzero]

@ulinuxp Thank you! Hopefully we'll soon have time to start with this...

[guysoft]

Here are some more libraries to consider:

* [Openface](https://cmusatyalab.github.io/openface/demo-2-comparison/)

* https://github.com/ageitgey/face_recognition

The latter is using Dlib state-of-the-art face recognition built with deep learning. The model has an accuracy of 99.38% on the Labeled Faces in the Wild benchmark

The exact same engine is used here in go
https://github.com/Kagami/go-face

[dtaivpp]

Going to look into starting this. Also, I think there should be a UI. Don't get me wrong I think that grouping people under a random tag would be less useful than allowing users to name their people.

Here are my thoughts for the logic and you all can let me know what you think. I haven' t looked too much over the code base but will do that before implementing this so it is integrated with native triggers.

Backend:

Application Starts
    if no models exist:
        start scanning all files and generating models
        models are randomly named and tags are added
    else:
        on new photo entry
        check against current models
        if no matching models
            create new random tag
        else
            update with random tag / or name if exists

Frontend:

Show Sample image with peoples faces and tag
    allow the user to change the tag to name
    on name entry, update model to use name to tag

I just wanted to put some sample logic out there so we can break this down into a few steps and get it progressively added. I think the first phase would be getting Go-Face creating models and grouping faces by random tag.

[dtaivpp]

Small update, it seems we will only need to tag the photos. I just noticed there is already support for re-naming tags. I'll start looking into just classifying and randomly naming them. Then users can simply update the labels. The tricky part will be photos with more than one person but we can cross that bridge when we get there.

[Gestas]

I've been testing the approach above vs requiring users to create a per-person. This is with go-face directly, no PhotoPrism integration at this point. I'm using my personal photos, ~18K images. Some with faces, others without. I'm getting a significantly better user experience with the latter approach. I'm not an expert in this space, perhaps I'm doing something wrong.

When I use go-face to automatically create models I end up with collections of faces 100's big. For adults the rate of incorrect matches can be as high as 20%, for children 45%. The user experience required to correct the model is poor, I have to remove images. If instead I label 20-30 images first the run the classifier I get better results with a significantly better user experience.

I suggest we prompt users to label a set of photos first then run the classifier. For images with unknown faces we could add an "unlabeled-face" tag that would allow users to easily find and tag more faces.

[dunderpate]

As a user, the approach of labeling a set of photos makes sense. Do we have a flow for re-labeling images that are incorrectly labeled, or to re-classify once the models are updated in order to correct mislabeled images? Thanks for the work on this, this project is getting really close to a viable Photos alternative once this is added, for my purposes at least.

[lastzero]

Labels are only meant for object classification, not for face recognition. We need to use a separate database table with different columns for this.

[dtaivpp]

@lastzero Can you explain to me why that is the case? I am new to this project and am not sure I understand the whole flow right now. Would adding names as labels interfere with other processes?

@Gestas # Edit, I see you don't have it integrated yet. Disregard this.# do you have a fork I could look at? I'm curious if the model may just need some fine-tuning. For example, we could put a higher threshold on what justifies a match and then as more data is added rerun and collect ones that previously didn't have a high enough confidence.

[guysoft]

You will likely need a "face id" and then that face id would link to information about the person (for example a name). That would require another table.

[lastzero]

Note you can easily reset and re-index faces if needed, for example after upgrading or when you've made a mistake that takes long to fix otherwise:

docker-compose exec photoprism photoprism faces reset -f
docker-compose exec photoprism photoprism faces index

Naming usually just takes a couple of minutes. With our new version released yesterday, you may even get better detection results with fewer false positives. Instead of executing the commands with Docker Compose, you may skip "docker-compose exec photoprism" and directly run the "photoprism faces ..." commands in a terminal wherever PhotoPrism is installed.

If you're happy with the results and didn't make a mistake you want to fix anyway, you don't have to reset existing people & faces. The new release is better at ignoring false positives, which leads to less noise and better performance (as less faces need to compared with each other). Also it enables you to detect smaller faces if you want to (see PHOTOPRISM_FACE_SIZE in our docs).

[simoncoul]

@lastzero would it be possible to get more documentation of the facial recognition parameters? Currently it does not indicate how the parameters influence the recognition.

[lastzero]

PHOTOPRISM_FACE_SCORE determines the quality of the face detection, lower quality means faces are less likely to be a face. You may experiment with this, we think the default is the best setting.

PHOTOPRISM_FACE_SIZE determines the minimum size of faces in pixels. You may lower this to 40 or 30 to detect smaller faces. Such small faces won't be used for clustering though, but they can be recognized based on existing clusters. Detecting more faces will reduce performance. You may experiment with this like we did for many weeks.

PHOTOPRISM_FACE_OVERLAP determines how much detected faces may overlap. You may experiment with this, we think the default is the best setting.

PHOTOPRISM_FACE_CLUSTER_CORE determines how many faces are needed to form a cluster core for recognizing people from individual faces. You may experiment with this, we think the default is the best setting.

PHOTOPRISM_FACE_CLUSTER_DIST and PHOTOPRISM_FACE_MATCH_DIST influence clustering and detection. See https://towardsdatascience.com/machine-learning-clustering-dbscan-determine-the-optimal-value-for-epsilon-eps-python-example-3100091cfbc to learn more. If you're not familiar with machine learning and math, you better don't play with it.

[lastzero]

You may also want to read this to get a better understanding how clustering works (explaining this in detail is out of scope for our docs, especially since we need to continue working on multi-user support now and don't have a lot of time): https://en.wikipedia.org/wiki/DBSCAN

[alexislefebvre]

PHOTOPRISM_FACE_SCORE determines the quality of the face detection, lower quality means faces are less likely to be a face. You may experiment with this, we think the default is the best setting.

The default value is 9, is it 9/9, 9%? I suggest we add the minimal and maximal values for PHOTOPRISM_FACE_SCORE to the doc:

https://github.com/photoprism/photoprism-docs/blob/0bba8a88d0668ed0970dad466a21692952d10e44/docs/getting-started/config-options.md?plain=1#L81

[lastzero]

I've spent a couple of hours yesterday improving our command and config flag descriptions (and also added two more options for you to play with, if you feel like this can lead to better results for your use case):

   --face-size PIXELS                       min face size in PIXELS (default: 40) [$PHOTOPRISM_FACE_SIZE]
   --face-score THRESHOLD                   quality THRESHOLD for faces (default: 9) [$PHOTOPRISM_FACE_SCORE]
   --face-overlap PERCENT                   face area overlap threshold in PERCENT (default: 42) [$PHOTOPRISM_FACE_OVERLAP]
   --face-cluster-size PIXELS               min size of faces forming a cluster in PIXELS (default: 80) [$PHOTOPRISM_FACE_CLUSTER_SIZE]
   --face-cluster-score THRESHOLD           quality THRESHOLD for faces forming a cluster (default: 15) [$PHOTOPRISM_FACE_CLUSTER_SCORE]
   --face-cluster-core NUMBER               NUMBER of faces forming a cluster core (default: 4) [$PHOTOPRISM_FACE_CLUSTER_CORE]
   --face-cluster-dist DISTANCE             similarity DISTANCE of faces forming a cluster core (default: 0.64) [$PHOTOPRISM_FACE_CLUSTER_DIST]
   --face-match-dist OFFSET                 similarity OFFSET for matching faces with existing clusters (default: 0.46) [$PHOTOPRISM_FACE_MATCH_DIST]

@alexislefebvre The face detection library is https://github.com/esimov/pigo, maybe you find more information there in the docs or code. I guess the score can go from 1 to 1000? No idea really, as those don't play a role in practice. That's either way too high or too low for good results. So you shouldn't use these values anyway. We've tested with many different pictures and decided to go for a min score of 9, which is dynamically increased for smaller faces. Machine learning often is more art than science.

Especially the similarity distances ("epsilon") for the face embedding vectors are super sensitive, so any significantly larger or smaller values won't work in practice. More information:

• https://dl.photoprism.org/research/A%20Comparison%20of%20Clustering%20Algorithms%20for%20Faces.pdf
• https://towardsdatascience.com/how-dbscan-works-and-why-should-i-use-it-443b4a191c80
• https://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html

[lastzero]

Continued revising configuration flag and command descriptions today, docs have been updated (noticed they contained a few obvious mistakes like a wrong default value for FFmpeg capture buffers, > 500 instead of 32):
https://docs.photoprism.org/getting-started/config-options/

[alexislefebvre]

@alexislefebvre The face detection library is https://github.com/esimov/pigo, maybe you find more information there in the docs or code. I guess the score can go from 1 to 1000? No idea really, as those don't play a role in practice. That's either way too high or too low for good results. So you shouldn't use these values anyway. We've tested with many different pictures and decided to go for a min score of 9, which is dynamically increased for smaller faces. Machine learning often is more art than science.

The author of this library answered to my question, I didn't understood everything but you may find it interesting.

[lastzero]

Thanks! In fact we've recently changed IoU in the stable version and it seems to have slightly improved results.

[grahamPegNetwork]

I just saw the comments here, way to go team! Docker pull, restart and look at it go! There has been a lot of activity since I last checked. Ultimately, who is most responsible for this feature happening? Is this the work of @lastzero ?