Process Frigate images with CompreFace

[jakowenko]

I love Frigate and have been spending the last few weeks trying to use it for a room presence detection system. I've been using Frigate + CompreFace + Node-Red to achieve this, but my Node-Red flow became pretty complicated and hard to maintain, so I decided to move my logic to a local API, which I containerized.

The API is exposed with a POST method /recognize which expects the MQTT frigate/events JSON as the payload.

Images are then processed though Facebox and/or CompreFace for identification.

The output from the API is an array of matching faces, which I then process through Node-Red to update the location for the matching users in Home Assistant.

Subscribing to the MQTT frigate/events topic directly is also supported and the matches faces are published to a different MQTT topic that can be defined by the user. To see all the available options, checkout the README.

[
    {
        "id": "1613103956.29759-4g05fi",
        "name": "david",
        "matched": true,
        "confidence": 62.91,
        "attempt": 1,
        "attempts": 1,
        "type": "latest",
        "time": 0.65,
        "camera": "office",
        "room": "Office",
        "totalTime": 0.68,
    }
]

Node-Red flow using the REST API

Node-Red flow subscribing to the topic that contains matches after being processed

I was hoping for feedback from the community and wanted to see if others would find something like this useful.

Github: https://github.com/jakowenko/double-take
Docker Hub: https://hub.docker.com/r/jakowenko/double-take

[pospielov]

Looks great!
How many frames per second did you achieve?
Could you share your thoughts about pros and cons of CompreFace and FaceBox?

[jakowenko]

I haven't had a chance to try that prediction_count flag yet on the issues I had when my girlfriend was over, but here's a strange false positive I had today.

Frigate detected my cat as a person, but CompreFace then detected my couch pillow as me with a pretty high similarity.

[pospielov]

Could you send me the original picture?
I tried to cut it from the screenshot and CompreFace didn't find any faces:

Also, look at the probability score - it's 0.76237, this is low for detection. In my experience even faces on bad quality photos have a probability of more than 0.9.
I looked at the code, looks like the default threshold is 0.65 now, probably need to change it in the next release :)
For now, you can use det_prob_threshold query param to change its default value.
The problem with the similarity still exists, don't understand how it could be so high...

[jakowenko]

Here's the raw photo. Thanks for the tip with the det_prob_threshold. I was looking at that last night, I'll try to use that with a value of .9 moving forward to see how the results look.

[pospielov]

So I checked your image and I can confirm, it detects pillow as a face :)
I made an experiment: I uploaded to a face collection 300 images of myself with different quality and named them as different subjects.
Then I uploaded this picture.
4 images of 300 crossed the threshold of 0.7 (I believe this is a threshold for face recognition), even for two of the images the similarity was more than 0.9 which is extremely high. All these 4 photos have awful quality.
Then I uploaded the image of you from your first post. As expected the similarity wasn't more than 0.52.

Then I made another experiment. I took 50000 photos of people from a public dataset and uploaded them to face collection.
And uploaded the photo with the pillow. There were 37 results with a similarity of more than 0.9. I checked the original photos and all of them have bad quality, mostly very dark.
Then I took the first photo that is "similar" to the pillow and uploaded it again to this 50000 photos collection. It returned a similar result with the pillow - 13 of 20 most "similar" faces were the same.
From this experiment, I can tell that the face recognition system works badly with faces with bad quality, especially for dark faces.

So what I would recommend you - upload faces with good quality to face collection. I think it could work better even if the test images are of bad quality.

[jakowenko]

Thanks for looking into this and reporting back your findings. I'll make sure to feed CompreFace better pictures of myself too. I've completely dropped using Facebox because CompreFace was working so well. Plus Facebox has a 100 image training limit, which is a bummer.

I do from time to time still get false positives between myself, my sister, and my girlfriend, but I'm hoping as I train the model with more images of each of us, then the results will improve.

[jakowenko]

I've made quite a few UI updates over the past few weeks and the root of the API at http://YOUR-IP:PORT will now serve a frontend that shows all matches and misses from the detectors. You can either retrain the models with the images or delete them. More updates to follow! Would love some feedback if anybody tries it out.

[pospielov]

I see from your screenshots that some of the results were made by CompreFace and some from Deepstack.
Do you use some ensemble of solutions to predict?
Like if both CompreFace and Deepstack predict that this is David, then you accept the result and if not, then do not accept?

[jakowenko]

I use both CompreFace and DeepStack currently and take the best result between them for each event that is processed.

Interesting idea though to maybe use the combination of both of them before confirming it's a positive match.

[jakowenko]

I pushed quite a few updates and small performance improvements last night, including support for CompreFace v0.5.0. If you are using multiple detectors, then the same image is passed on each processing loop. This allows me to display a single image on the UI with the results per detector.

I've also added some more documentation on how I setup my Node-Red and Home Assistant notifications.

More notes on what's included in this release can be found here.

[jakowenko]

More updates 🚀

CompreFace has definitely been the best performer so far, so thank you again for the amazing work!

• I've migrated the ENV variables to a more readable config.yml file.
  Here's a small sample:

  server:
    port: 3000
  
  mqtt:
    host: 192.168.1.1
  
  home_assistant:
    url: http://192.168.1.1:8123
    token: xxx.xxx-xxx
  
  confidence:
    match: 60
    unknown: 40
  
  save:
    matches: true
    unknown: true
  
  frigate:
    url: http://192.168.1.1:4000
  
  detectors:
    compreface:
      url: http://192.168.1.1:8000
      key: xxx-xxx-xxx-xxx-xxx
    deepstack:
      url: http://192.168.1.1:8001

• Added Home Assistant support. Entities for people and cameras will automatically be created

Any feedback or feature suggestions would be appreciated.

[pospielov]

Hi, sounds great!

I visited your repository to understand it better and probably give some feedback.
And here is the problem - I haven't used Frigate, Home Assistant, and Node-Red.
For me, it doesn't clear what exactly your solution does.
"Unified UI and API for processing and training images for facial recognition." - sounds exciting, but then I read about your use cases and don't get it. I mean I assume I can't just use it to train CompreFace face recognition service, without Frigate or Home Assistant, right?
So what I suggest:

1. Be more clear in the description - for whom this solution? E.g. Frigate users that want to add face recognition.
2. The requirements, e.g. Frigate version XX
3. I would be really happy if there was a video demo with installation and first test (I think we need it in CompreFace too :) )

[jakowenko]

This is great feedback, thank you!

I agree, I need to better explain some of the use cases for people who don't use any of those third party applications.

[jakowenko]

1.0.0 is officially out with support for web sockets, pagination, updated filtering, and more!

The configuration section of the README is also updated with better examples.

Thank you to everyone who has used and help test Double Take over the past few months and for all the awesome feature suggestions. Keep them coming!

[pospielov]

Wow, congratulations!
BTW your documentation is much better now! I really like that you published Postman documentation. We also will publish it very soon :)

[jakowenko]

Thank you! It's been a journey over the last 6 months haha. I'm telling everybody I know about CompreFace, keep up the amazing work! It's been fun to update my project as you add more features.

[jakowenko]

v1.4.0

Bug Fixes

• api: validate mime type and result of database.get.fileByFilename
• api: catch facebox error before normalizing data
• api: delete orphaned db records when deleting training folder / files
• ui: fix multiselect dropdowns when open and scrolling
• ui: theme wouldn't load if setting to same theme

Features

• support for ARMv7 architecture
• api: api/latest/<name>.jpg
• api: use camera name when using custom mqtt topic
• ui: enable/disable sockets on matches page

[pospielov]

A defined roadmap is quite difficult as our progress depends on what contributors do we have.
But I understand how it's important, I think I'll add something to the readme.
The main ideas to improve CompreFace (the order may be random, depending on the contribution skills):

1. Add UI to manage face collection
2. Create or find a new face recognition model, based on a dataset with better distribution (gender, nationalities, etc.)
3. Improve the performance. Now we just reuse the most popular face recognition libraries, but they are not optimized for performance.
4. Add support of ARM devices and TPU accelerators, like Google Coral
5. Integrate with the most popular solutions like Home Assistant, BlueIris, Shinobi.
6. Add much more examples and videos

train false positives or tell CompreFace that it got the recognition wrong.

This is technically impossible, we take the face recognition model and use it for all recognitions.
You still can retrain this model by yourself and then build CompreFace using your model. Probably we can do something to simplify this process.

[alexdelprete]

This is technically impossible

Pospielov, can you explain for me why it's technically impossible? I'm an old-school dinosaur, and in the old days of bayesian filters to detect email spam, negative training (telling the filter it detected wrongly a legit email) was critical in order to improve and refine its efficiency. Many Double Take users (I was one of the first asking David about this) are asking why we can't tell the engine that it was a false/wrong detection thinking it can "learn" from its mistakes.

I guess we're not yet in AI with CompreFace, right? :)

Thanks for your amazing work and most of all, your support to the community, it's great how you support the solution and are open to discussing it to improve it, thank you for this, it's highly appreciate it.

[pospielov]

https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78
Here is a good article to understand how face recognition works. Of course, there were lots of improvements from 2016, but the main idea remains the same:

1. You take a huge dataset and train your huge convolutional neural network. It could take days even on a powerful GPU.
2. Then you remove fully connected layers on this CNN, so there is a vector of 512(or 128) numbers at the end of the neural network instead of the answer who is this person
3. Then you receive this vector for each saved face (I believe this is called training in double-take, but in CompreFace we just collect vectors, no ML here, so no training)
4. When you need to recognize a face, you calculate the vector for an unknown face, then find the euclidian distance (linear algebra, again without ML) between the unknown face and all other faces. This distance answers the question of whether two faces look similar.
   So, in the result application, there is no ML or training, if the CNN model is bad, we can't improve it.
   Of course, technically it's possible. But the training is not so easy process, there are lots of things to consider - how many layers to train, what training coefficients and iterations. And it depends on the goals and number of examples. I believe if you know all this stuff, you can make it with the script(you can find them in FaceNet or InsightFace libraries) and then just put it into CompreFace. I already thought if we need to add the functionality of uploading a new model from UI. But I believe it won't be a popular feature.

[alexdelprete]

4. When you need to recognize a face, you calculate the vector for an unknown face, then find the euclidian distance (linear algebra, again without ML) between the unknown face and all other faces. This distance answers the question of whether two faces look similar.

So basically the vector represents a physical measurement of specific parts of a face, a sort of digital print of the face, and when you scan a new face and take its vector, you search in the neural network for a close enough vector?

I will read the article, thank you very much.

[pospielov]

I use a neural network to get a vector. Then just calculate euclidian distance to find the similarity

[jakowenko]

I've recently published version 1.12.0 with includes support for AWS Rekognition and I wanted to just say that CompreFace is just as good if not better. Hats off to you guys for providing such an amazing open source project. I wanted to get Rekognition added just so I could test it side by side the other detectors. We'll see over time how it does, but I prefer the price of CompreFace 🍺

1.12.0 (2022-06-13)

Features

• api: opencv preprocess face check (ed30ad1)
• aws rekognition support (7904852)
• detectors: process images from specific cameras (5d39d0c)
• frigate: sort sub labels alphabetically #217 (82d8736)
• frigate: stop_on_match config option to break process loop (4b98990)
• opencv: adjust classifier settings via config (2e6c512)
• ui: show config errors (ddcaf89)
• ui: upload images to process with detectors (f774406)

Bug Fixes

• api: sleep if image hasn’t changed during retry loop (78e9808)
• better handling of update check (6de1cf1)

[pospielov]

I am happy that your project continues to develop.
Just curious, as I understand, your solution sends requests for recognition only if Frigate founds the face, right? So it shouldn't be so many requests per month. Do you know how many recognitions per month you do?

Also, api: opencv preprocess face check ([ed30ad1](https://github.com/jakowenko/double-take/commit/ed30ad10dfe9f6d104619fa134e2d54698b1c1ff)) - did you add another face detection? But why?

[jakowenko]

Thank you @pospielov!

Double Take is triggered when Frigate detects a person, so there is a chance a face might not be in the frame when passing the image to the detector. And because DT pulls frames to process, I thought it would be a nice touch to run it through OpenCV if the user wants.

I have 6 cameras setup within my house, so running Rtekognition on all of those would probably start to add up haha. I managed to run up around 5,000 requests with 2 cameras over the course of a few days.