Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

People: Facial Recognition #22

Open
lastzero opened this issue Oct 8, 2018 · 113 comments
Open

People: Facial Recognition #22

lastzero opened this issue Oct 8, 2018 · 113 comments

Comments

@lastzero
Copy link
Member

@lastzero lastzero commented Oct 8, 2018

Issuehunt badges

As a user, I want to tag familiar human faces so that I can find all photos of these persons.

PhotoPrism must be able to detect and recognize faces. There are existing libraries for this, so we should try those first. See Face Recognition in our Wiki.

Here is a complete howto: https://hackernoon.com/face-recognition-with-go-676a555b8a7e

Acceptance Criteria:

  • You SHOULD use go-face (unless you find a better library)
  • A unit test demonstrating face recognition with real images and faces MUST be delivered
  • A user interface MAY be developed and gives extra karma
  • You MUST update our Docker images in /docker as needed (go-face requires dlib as external dependency); let us know when we shall push new images to Docker Hub
  • Documentation MUST be updated so that other developers know everything to continue working on this task

IssueHunt Summary

Backers (Total: $640.00)

Become a backer now!

Or submit a pull request to get the deposits!

Tips

@husayt
Copy link

@husayt husayt commented Oct 15, 2018

Here are some more libraries to consider:

The latter is using Dlib state-of-the-art face recognition built with deep learning. The model has an accuracy of 99.38% on the Labeled Faces in the Wild benchmark

@lastzero
Copy link
Member Author

@lastzero lastzero commented Oct 15, 2018

I'd prefer a library implemented in Go to improve performance and reduce complexity.

@devzsolt
Copy link

@devzsolt devzsolt commented Oct 19, 2018

As a proof of concept you can create a small interface to an external tool (eg. https://github.com/ageitgey/face_recognition using its CLI) and then later it's possible to replace it with a native Go lib. I think it would speed up the early development, this Python lib looks pretty much mature.

@IssueHuntBot
Copy link

@IssueHuntBot IssueHuntBot commented Dec 19, 2018

@issuehuntfest has funded $40.00 to this issue. See it on IssueHunt

@Xeoncross
Copy link

@Xeoncross Xeoncross commented Apr 10, 2019

The easiest way to do this is simply to slap https://github.com/esimov/pigo into the project as a pure Go include.

@lastzero
Copy link
Member Author

@lastzero lastzero commented Apr 18, 2019

@Xeoncross Thanks! But Pigo is just face detection, not face recognition...

See https://github.com/photoprism/photoprism/wiki/Face-Recognition

@lastzero
Copy link
Member Author

@lastzero lastzero commented May 4, 2019

This looks good to me on first sight: https://github.com/jdeng/goface

@trosel
Copy link

@trosel trosel commented Nov 17, 2019

Could object detection be included in this request as well? Some libraries work with both faces and objects I think.

@lastzero
Copy link
Member Author

@lastzero lastzero commented Nov 21, 2019

@trosel We already have object detection / classification, without position however. Facial recognition is different from classification in that you need parameters for face parts as output to do the actual recognition based on a clusters (and not just detection).

What specific library do you refer to?

@cgudea
Copy link

@cgudea cgudea commented Dec 5, 2019

I don't have any go experience but I'd really like to see this feature implemented. Is there anyone currently doing work on this? If so, would adding money onto the bounty help expedite the development?

@lastzero
Copy link
Member Author

@lastzero lastzero commented Dec 5, 2019

@cgudea You're most welcome to support our work via GitHub Sponsors:

https://github.com/sponsors/lastzero

No contributions there yet as GitHub Sponsors is pretty new. They will match any donation in the first year!

Don't expect to see a pull request for facial recognition anytime soon, it's too complex. We'll start working on it once albums and sharing are done. Financial support makes a major difference as we otherwise have to stop development frequently to do commercial projects.

@cgudea
Copy link

@cgudea cgudea commented Dec 5, 2019

@lastzero totally understandable. I'm a huge fan of the goals of this project and will be looking to contribute after the holidays in January either financially or via development (I am a mobile developer at my dayjob).

@lastzero
Copy link
Member Author

@lastzero lastzero commented Dec 5, 2019

@cgudea Thank you <3

@ghost
Copy link

@ghost ghost commented Dec 7, 2019

@lastzero If you want to get rid of the dependency of dlib, you can try the way that ownphotos does facial clustering. https://github.com/hooram/ownphotos/blob/dev/api/face_clustering.py

You can use any tensorflow CNN model that extracts facial features, and do a kmeans on them with https://github.com/muesli/kmeans

@lastzero
Copy link
Member Author

@lastzero lastzero commented Jan 16, 2020

@ulinuxp Thank you! Hopefully we'll soon have time to start with this...

@guysoft
Copy link
Collaborator

@guysoft guysoft commented Jan 23, 2020

Here are some more libraries to consider:

* [Openface](https://cmusatyalab.github.io/openface/demo-2-comparison/)

* https://github.com/ageitgey/face_recognition

The latter is using Dlib state-of-the-art face recognition built with deep learning. The model has an accuracy of 99.38% on the Labeled Faces in the Wild benchmark

The exact same engine is used here in go
https://github.com/Kagami/go-face

@dtaivpp
Copy link

@dtaivpp dtaivpp commented Feb 25, 2020

Going to look into starting this. Also, I think there should be a UI. Don't get me wrong I think that grouping people under a random tag would be less useful than allowing users to name their people.

Here are my thoughts for the logic and you all can let me know what you think. I haven' t looked too much over the code base but will do that before implementing this so it is integrated with native triggers.

Backend:

Application Starts
    if no models exist:
        start scanning all files and generating models
        models are randomly named and tags are added
    else:
        on new photo entry
        check against current models
        if no matching models
            create new random tag
        else
            update with random tag / or name if exists

Frontend:

Show Sample image with peoples faces and tag
    allow the user to change the tag to name
    on name entry, update model to use name to tag

I just wanted to put some sample logic out there so we can break this down into a few steps and get it progressively added. I think the first phase would be getting Go-Face creating models and grouping faces by random tag.

@dtaivpp
Copy link

@dtaivpp dtaivpp commented Feb 25, 2020

Small update, it seems we will only need to tag the photos. I just noticed there is already support for re-naming tags. I'll start looking into just classifying and randomly naming them. Then users can simply update the labels. The tricky part will be photos with more than one person but we can cross that bridge when we get there.

@Gestas
Copy link

@Gestas Gestas commented Feb 25, 2020

I've been testing the approach above vs requiring users to create a per-person. This is with go-face directly, no PhotoPrism integration at this point. I'm using my personal photos, ~18K images. Some with faces, others without. I'm getting a significantly better user experience with the latter approach. I'm not an expert in this space, perhaps I'm doing something wrong.

When I use go-face to automatically create models I end up with collections of faces 100's big. For adults the rate of incorrect matches can be as high as 20%, for children 45%. The user experience required to correct the model is poor, I have to remove images. If instead I label 20-30 images first the run the classifier I get better results with a significantly better user experience.

I suggest we prompt users to label a set of photos first then run the classifier. For images with unknown faces we could add an "unlabeled-face" tag that would allow users to easily find and tag more faces.

@dunderpate
Copy link

@dunderpate dunderpate commented Feb 25, 2020

As a user, the approach of labeling a set of photos makes sense. Do we have a flow for re-labeling images that are incorrectly labeled, or to re-classify once the models are updated in order to correct mislabeled images? Thanks for the work on this, this project is getting really close to a viable Photos alternative once this is added, for my purposes at least.

@lastzero
Copy link
Member Author

@lastzero lastzero commented Feb 25, 2020

Labels are only meant for object classification, not for face recognition. We need to use a separate database table with different columns for this.

@dtaivpp
Copy link

@dtaivpp dtaivpp commented Feb 25, 2020

@lastzero Can you explain to me why that is the case? I am new to this project and am not sure I understand the whole flow right now. Would adding names as labels interfere with other processes?

@Gestas # Edit, I see you don't have it integrated yet. Disregard this.# do you have a fork I could look at? I'm curious if the model may just need some fine-tuning. For example, we could put a higher threshold on what justifies a match and then as more data is added rerun and collect ones that previously didn't have a high enough confidence.

@guysoft
Copy link
Collaborator

@guysoft guysoft commented Feb 25, 2020

@Gestas
Copy link

@Gestas Gestas commented Feb 26, 2020

@dtaivpp I think @lastzero is right, we need to be able to differentiate between labels (nouns) and faces during a search. E.g in English "Forest" can be a set of trees or a name for a human. A user might want to search for "Forest" the person and not a set of trees.

@el-tiuri
Copy link

@el-tiuri el-tiuri commented Feb 15, 2021

If importing previously tagged faces is something you'd want to implement (I think it'd be great!) importing from XMP metadata should probably be doable. It defines an area, coordinates and the name associated with the face. I attached an XMP file (renamed to txt so I could upload it here) with a person tagged. To anonymise it I changed the name to "Dummy data".

DSCF1911.JPG.txt

@lastzero
Copy link
Member Author

@lastzero lastzero commented Feb 15, 2021

@el-tiuri For this, we need a proper XML / XMP parser first. The one that comes with Go is "not so good" and currently being rewritten from what I know. Once all this work is done, it may be easy to import faces from XMP.

@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Feb 15, 2021

@conrad82 has funded $5.00 to this issue.


1 similar comment
@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Feb 15, 2021

@conrad82 has funded $5.00 to this issue.


@pluja
Copy link

@pluja pluja commented Feb 22, 2021

I know you want to use a written-in-go library, but maybe this project I just found might help somehow. So I will leave the link to the repo just in case:
https://github.com/exadel-inc/CompreFace

@lastzero
Copy link
Member Author

@lastzero lastzero commented Feb 22, 2021

Interesting! So many options... was different when we started... patience pays out :)

https://github.com/esimov/pigo received major improvements as well... I like the simplicity and that it doesn't have external dependencies (as we already have enough to maintain).

@alexislefebvre
Copy link
Contributor

@alexislefebvre alexislefebvre commented Feb 22, 2021

Have you considered using a separate Docker image for image detection? Maybe it already exists and it can be plugged through an HTTP API, it would avoid managing the dependencies.

@lastzero
Copy link
Member Author

@lastzero lastzero commented Feb 22, 2021

Decided to become @esimov's first GitHub Sponsor 🥳 🎈

He's maintaining Pigo and a bunch of other libraries that look amazing. Can't wait to give them a try!

yeah

@esimov
Copy link

@esimov esimov commented Feb 22, 2021

@lastzero thank you, I'm really grateful for your generosity and also for your kind words. 🙏 Also keep in mind that Pigo is still in active development and there is still one big thing left behind in the upcoming future: object recognition and description.

@lostmaniac
Copy link

@lostmaniac lostmaniac commented Mar 11, 2021

How long can I expect to use this feature? Looking forward to face recognition

@thomaseizinger
Copy link

@thomaseizinger thomaseizinger commented Mar 11, 2021

How long can I expect to use this feature? Looking forward to face recognition

It is gonna be faster if you join the list of sponsors :)

@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Mar 12, 2021

An anonymous user has funded $20.00 to this issue.


@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Mar 14, 2021

@angelnu has funded $20.00 to this issue.


@dror3go
Copy link

@dror3go dror3go commented Mar 29, 2021

Is there a plan to support pets recognition, or is it out of the scope of this issue?

@graciousgrey
Copy link
Member

@graciousgrey graciousgrey commented Mar 30, 2021

@dror3go it is out of scope of this issue. But it is something we would love to have as well at some point :)

@AndreyGurevich
Copy link

@AndreyGurevich AndreyGurevich commented Mar 30, 2021

Anyway, it will be a great success if the black cat will be recognized as a cat, not as a "dark area of the photo" :)

@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Apr 2, 2021

An anonymous user has funded $15.00 to this issue.


@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Apr 20, 2021

An anonymous user has funded $5.00 to this issue.


@issuehunt-app
Copy link

@issuehunt-app issuehunt-app bot commented Apr 27, 2021

@bitbacchus has funded $5.00 to this issue.


@praul
Copy link

@praul praul commented May 10, 2021

If anyone is interested in a good workaround until the feates comes native to photoprism:

  • Use Digikam for Facial Recognition and Tagging. Set Digikam to write People in pictures metadata. You can use https://github.com/mikenye/docker-digikam for headless digikam
  • Photoprism reads the peoples tag as "image content". You can search for persons over the searchbar
  • :)
@guysoft
Copy link
Collaborator

@guysoft guysoft commented May 10, 2021

@praul Thats cool - if you want to help me add it to PhotoPrismPi I think it would be a worth addition.

@dada051
Copy link

@dada051 dada051 commented May 12, 2021

I have already tagged photos (from the Windows Live Photo Gallery) that digiKam reads, so I think it's OK. But I have no results when searching on PhotoPrism. Any idea ?

@graciousgrey
Copy link
Member

@graciousgrey graciousgrey commented May 12, 2021

@dada051 PhotoPrism might read other fields than digiKam. You find an overview here: https://docs.photoprism.org/user-guide/advanced/metadata/. Do you know to which metadata field Windows Gallery writes the tags?

@dada051
Copy link

@dada051 dada051 commented May 12, 2021

I'm assuming that digiKam only reads face tag from the standard XPM tag
image
I don't know if it's standard, but it's from this tag that digiKam found the face

@graciousgrey
Copy link
Member

@graciousgrey graciousgrey commented May 12, 2021

@dada051 If you send us an example xmp file we might be able to add the field

@dada051
Copy link

@dada051 dada051 commented May 12, 2021

Thanks, but I don't have XMP sideloaded. But the document here explain the tags: https://docs.microsoft.com/en-us/windows/win32/wic/-wic-people-tagging

I will try to sideload an XMP to send

@TYKUHN2
Copy link

@TYKUHN2 TYKUHN2 commented May 12, 2021

I've already implemented go-face natively within Photoprism, unfortunately I hate HTML (and Vue) coding and before I could finish it I got distracted by family and school matters. I suppose I can release a stop-gap measure that is incomplete, but one of the core features (training) would be limited. I also have not tested the implementation yet, because I did not finish the UI, and I did not feel like simulating REST commands at that point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Roadmap 🚀✨
Development 🐝
Linked pull requests

Successfully merging a pull request may close this issue.

None yet