Custom Models

[blakeblackshear]

I have been thinking about the best way to address custom models and decided that I will be building a companion service for Frigate to send examples to for training. I plan to build this first for the few camera systems that I manage, and then open it up to others in a limited beta. It will be a hosted service and it will be implemented in a way that requires you to explicitly configure the URL where examples are sent in your configuration. It will obviously be disabled by default.

Things I expect to be able to do with a robust set of examples:

• Train an object set that is more relevant for security cameras (people, car, truck, fedex, ups, packages, deer, bird, face, license plate, bicycle, etc) and remove things in the standard COCO data set that aren't needed
• Optimized models for IR (night vision) vs daytime
• Optimized models for camera height and/or location (front door, eye level, second story eave, etc)
• Test trained models against historical examples (from your cameras and others) to determine accuracy

Some points of clarification:

• This service isn't something that I could provide for free. I personally don't want my examples in a public database and storage and cloud services cost money.
• I want to aggregate examples across many different systems to avoid over fitting the model to myself and my family. This also enables models to be evaluated against other people's examples.
• I know that most of the community here is very privacy conscious. Your examples would be used to train and evaluate models, but never shared with anyone else. The terms and conditions will allow me to use your examples for training/evaluating models only.
• Frigate itself will always be free and open source. It will always work with off the shelf or your own models too. This will be entirely optional.
• The way examples are sent to Frigate will be very transparent, and you are welcome to build your own endpoint and have Frigate send examples there too.
• I expect the cost for this to be somewhere in the ballpark of $5/mo. I may offer a discount for those contributing to the model vs those just using it.

I would be grateful for feedback/questions below. Either way, I intend to build this for my own personal use.

[savikko]

I applaud this development. I happily pay fee for this.

[dziobson]

Me too

[thelefty]

Happy to beta test it for you. Sounds exciting

[Vincentig]

Definitely ready to help !

[biggen1684]

I also welcome this. License plates are really exciting. I use OpenALPR with a dedicated license plate camera currently, and it does work but it requires a constant connection with their servers. Would love the ability to be able to do this without a cloud connection in the future.

I have no problem providing you with license plate models. I get about 3100 plates a week currently.

[Kate222]

I am interested in viewing license plates, especially for speeding cars.

[ThreepE0]

what specifically would you do with that information? It'd be illegal to dox and contact them (you'd open yourself up to all sorts of legal action, and rightfully so.) And, police couldn't follow up based on any information you give them. The best they could do would be put out speed traps or collect data themselves if you have people frequently speeding by.

[bagobones]

You can provide the related video, repeat pattern of time of day.

You provide the video then the police follow up if they think it is worth it and try to catch them in the act, and then include the video as additional evidence of the behaviour.

The plate tagging just makes it easier to search.

[zeliant]

I'm am willing to contribute and also subscribe if it gives me better detection accuracy

[stryker007]

being as training generally speaking requires a variety of images and more the better I think its a great idea that we club together for this... the running costs sound very reasonable. Am I right in saying we're suggesting frigate automagically send captured images for this? So long as I can specify which cameras sounds fine!

[pjfian]

Sounds great. Look foward to this.

[ronschaeffer]

Very happy to beta test

[digiblur]

Sounds awesome and I don't have any issues with things. Glad to see you explained a lot of the concerns before they were even brought up.

[danielchalef]

Rather than centrally aggregating samples have you considered using federated learning? I haven't dug too deeply, but it appears there have been a number of recent papers written regarding federated learning for object detection:
https://dl.acm.org/doi/10.1145/3387168.3387181
https://arxiv.org/pdf/2001.06202.pdf

[blakeblackshear]

I have seen some of the federated learning research. That may very well be the future for this type of thing, but I don't want to wait for it to mature. There is much better tooling built around the existing stuff.

[danielchalef]

Makes sense.

…

On Thu, May 6, 2021 at 2:55 PM Blake Blackshear ***@***.***> wrote: I have seen some of the federated learning research. That may very well be the future for this type of thing, but I don't want to wait for it to mature. There is much better tooling built around the existing stuff. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1043 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAAZ555TEKM5JZORNDC6DTMMF6BANCNFSM436JF6WQ> .

[ElMoribond]

Ultimately, a model specific to video surveillance will be of value.
Because if it proves its effectiveness, it will be sold (one way or another).
I am talking about the model and not the images used to create it.

The idea of funding the creation of this model by users is understandable.

I would love to, but I can't afford $5/month.
But I can spend time validating (or invalidating) each discovery and sharing.
Or, I didn't understand, all detections are sent as true?
In short, Ok to participate and share, but not to pay.

Imagine ... 2 years from now.
You sell the model to trucmuch.com
Will you donate part of the sale to people who subscribe to your service?
If so, what will be the terms? (duration of the subscription ?, Number of images shared ?, Amounts paid?, ...?)

[blakeblackshear]

I agree it will have value. How much, I don't know given how easy it is to I don't have all the answers to your questions. I am thinking of ways to enable those contributing to earn a discount or indefinite access to the model.

I will likely think of other services surrounding frigate in the future. Just like anything else, if something isn't worth it's price tag, people aren't obligated to pay for it.

[kdill00]

I agree it will have value. How much, I don't know given how easy it is to I don't have all the answers to your questions. I am thinking of ways to enable those contributing to earn a discount or indefinite access to the model.

I will likely think of other services surrounding frigate in the future. Just like anything else, if something isn't worth it's price tag, people aren't obligated to pay for it.

Also, I currently run Frigate in one of my businesses that is a large trampoline park on 26 cameras. 1080p at 8fps. Using one single PCI coral and one of the new double tpu ones. I have signs saying everyones on camera because people lose stuff all the time, so I would assume as long as it were anonymous, i could assist with working on models. Low light detection is one area i would love to foster.

[tubalainen]

Im Game!

My system here at home is now built around Frigate! The 0.8.4 release has been so stable for me. Just due to conf changes I needed to restart the docker.

[Tordux]

Hello,
It will ge good, but is it possible to train it localy with just our own pictures captured with our cameras ?
As I am not an expert, I an not sure if it will be more accurate or not.

It will be good to have possibility to add our own labels. For exemple detecting if it is my car or an intruder car. or if it fedex or ups, etc ...

[gpete]

This is fantastic! I've been using Frigate for a little while now and have arrived at the point where I too am wanting to hone in on the accuracy and expand detection to detect more useful objects. The coco models are obviously not trained with security cameras in mind and have pretty terrible accuracy at night vision in my setup. I've have been gearing up to take it on and create my own model by learning about training and retraining. I'd love to be able to extend my growing set of automations to detect certain service vehicles (mailman, garbage truck, delivery trucks).

Currently I'm trying to collect images of object of interest, false positives, and night vision footage to get enough input to train on. Working with more images to try and address that weakness. As @blakeblackshear pointed out the more images you have from other sources means you'll have a better model for general use. I've used Sighthound for many years and this is exactly the course they went down too. They went through a phase where night footage (IR illuminated) in particular had issues so they added a reporting feature in their application where users could elect to submit footage of false positives that were then sent to their servers to refine their models for future releases. They got us users working for them to collectively build a better classification model and it worked rather well. I'd love to see this for Frigate too.

I'm willing to help development wise. I have Python development experience along with some limited AWS and Tensorflow experience. I do also realize the desire to provide this as a closed source addition to compliment the already stunning Frigate project and respect that course too (much like Nabu Casa is to Home Assistant).

[dopeytree]

Would love to see integration of more of the already available google models like birds which are used in 'who's at my feeding table' https://github.com/mmcc-xx/WhosAtMyFeeder

Then refinement on things like numberplate detection or recording, fire, smoke, water.

My number one thing is needing everything to work at night. Currently it's all a bit pants at night mainly due to camera quality changes alot at night. Am toying with using a larger IR floodlight to help with night imagery.

[bagobones]

Using the NEW sound detection, looking just for the LOUD sound would probably be enough. However I believe there are specific tools for detecting "smoke detector" in audio as a specific type of audio.. Over all this might be more reliable than visual detection.

[NickM-27]

Fire alarm is a type of audio that is detectable with the smart audio detection coming in frigate 0.13

[dopeytree]

Just trying to work how to install and set this up on unraid. I have frigate working in a docker container. Hoping someone can make an unraid template for @whosatmybirdtable

[wranglatang]

Ive just spotted "whosatmybirdtable" in the apps of UNRAID! installing now!!

[dopeytree]

As now have this bird detector working I assume it's fairly easy to add other models too in parallel like this? For other stuff like licence plate/s. Would be cool to add in a general garden wildlife one. For example my dad has hedgehogs in his garden at night which would be cool to see.

[dz4news]

I am having a hard time understanding what this actually means... "Objects in locations you want to avoid are not false positives"

Looking at the screen capture ... do I tag my dog as a "false positive" since it was tagged as a bird?

[NickM-27]

You would mark this as this is not a bird and then go into frigate+ and label it as a dog

the text Objects in locations you want to avoid are not false positives is explaining that if you only care about cars in your driveway but the snapshot shows a car in the street -- this is not a false positive. The only time something is a false positive is when it is labeled as something that it is not

[bagobones]

Maybe

"False positive reports should only to be used to report objects with incorrect names / labels)" for all other issues please check the documentation or file a support ticket.

[ccutrer]

So one thing I'm running in to:

I have a couple locations that are perennial false positives as people (coats hanging on a rack, laundry draped over a chair, blankets draped over a railing). My problem is that you cannot actually submit to Frigate+ while an event is in progress. But the event is always in progress. I've tried turning on/off lights, or walking in front of the camera to obstruct the false positive just to end the event so I can submit it. Sometimes this works, but often it does not, or worse (for purposes of training the model) the true positive (me!) just becomes the snapshot for the event when it ends, making it useless for training as a false positive. Would it be possible to either a) submit to Frigate+ while an event is in progress (using the latest snapshot from the camera), or b) manually "end" tracking of a stationary object, concluding the event, so I can then submit it.

[bagobones]

Fairly sure the event will end cleanly if you restart frigate from the menu, as a workaround

[ccutrer]

It shows up in events if you do that, but there's no snapshot, so it just fails trying to submit to Frigate+

[NickM-27]

if you restart frigate the image will not be saved to the file system. This is also why in progress events can't be submitted.

This is most likely something that can be addressed in some way in 0.14 with the new UIs that are planned including the ability to submit to frigate+ from the recording view

[slopokdave]

What should I be marking as "FedEx", "amazon", etc? The person? The Vehicle? Should I mark them as BOTH fedex and a car? Same with a package, should I mark it as "amazon" and "package?" Thanks. :)

[NickM-27]

this is answered in the docs https://deploy-preview-6262--frigate-docs.netlify.app/plus/#properly-labeling-images

amazon, fedex, and ups refer to the logo only. package refers to the box that is delivered

[slopokdave]

Thank you, missed that section!

[ehn]

this is answered in the docs https://deploy-preview-6262--frigate-docs.netlify.app/plus/#properly-labeling-images

amazon, fedex, and ups refer to the logo only. package refers to the box that is delivered

What isn't 100% clear from the docs is whether we should also label logos on things other than trucks, such as packages and uniforms.

[Eiritj]

Hi. I'm using Frigate as a add-on in Home Assistant (Generic x86-64 install on a PC with a PCI Coral), and I have a case where I think Frigate+ might be the solution for me. Basically I want to be able to distinguish between my cat and the neighbours cat so that I can lock/ unlock the pet flap based on which cat is detected. For this I understand that I need a custom model.
I already have a bunch of snapshots from Frigate of both cats that I think can be used for training. The questions I have are:

1. Is this a case and use where Frigate+ is suitable and capable?
2. Can Frigate+ be installed in my case?
3. Is this extremely technical to do? Are there tutorials/ guides available?

[Eiritj]

Thanks for the suggestion. I am already looking into cat doors that use RFID (DIY and ready products) since the cat is chipped, but they have several trade-offs. The major ones being range, integration with HA, and unreliable readings if not using batteries to power them. The easiest solution is a flap like the SureFlap combined with a door/windows sensor, but that leaves me with two more units that need new batteries every once in a while.
Other than that RFID seems to be the way to go. I lack the skills required for image analysis.
Perhaps a person could be hired to create what I need for image analysis, but even then there would be a question of support and updates, so it would not be feasible to outsource it.

[interbiznw]

Yeah seems more feasible to hire someone to make a home assistant addon for a door such as the Pawport, collar based seems like they would have better range than trying to read pets rfid chip.

[Eiritj]

Collar based methods open up a lot of possibilities, but also come with drawbacks.
In general, collars are not advisable for cats. The cat brain is very poor at computing events like "collar stuck in branch", and easily interprets this as "I am being eaten". This overloads the cat and can cause it to harm itself needlessly.

[jannehietamaki]

Another product here https://www.flappie.ch/en/

[dopeytree]

I'm using basic frigate to track my cats using the google cat tag. I have tagged cat & dog as sometimes cat doesn't always work.

I have dedicated cameras placed with a view of each cat flap. Then some generally around the garden so I can pinpoint roughly the last direction. Sure it doesn't work if the cat goes out the door or a window (they are outdoor cats) but I find it helps. For some reason cats are never picked up on the drive camera which is at about headlight. The rest are all on the floor while testing. They are annke c500 POE cameras so only one cable to run and then I use POE extenders ( a small self powered switch that takes 1port and gives you 5x ports).

I also tried building an RFID tag on the catflap it also needs a motion sensors to detect direction of travel. But honestly I think the best product would be a BLE beacon and then use a few ESP BLE proxies around the house and garden to pinpoint location but I gave up at that point too much error plus my cat didn't like his apple tag and it made his fur fall out around the collar area. so frigate works best. Cameras must be low to floor for best detection.

[solidstudio]

Has anyone tried these models

https://github.com/MikeLud/CodeProject.AI-Custom-IPcam-Models/tree/main/TensorFlow%20Edge%20TPU%20Models/assets

[slopokdave]

Any way for me to submit to Frigate+ and tell it this is NOT FedEx? :) Thanks, (Note: this was before I raised my thresholds, but still would be good if I could "correct" my model)....

edit: Now that I think about it, should I submit it but do not create the fedex box?

[ccutrer]

Yes. Submit it, label the car, but do not label FedEx, UPS, or Amazon.

[gdh123]

I have set up a test environment for frigate, just to compare it with other solutions on the market and I do get a lot false positives or no event in case someone just rushes thru the cam picture.

so I signed up for frigate + (free), uploaded some pictures but it seems the other models are not available. the help button does not work for me, so I thought I could shootout the question on where I can find some guideline to train my own model?

maybe the positioning of my cam is not optimal for the majority of other users, so the existing models do not work properly for me.

[NickM-27]

I'd suggest reading #7932

[interbiznw]

Is there no way to identify USPS Truck/Mail van?

[MickPBduece]

There is a market for this sort of thing including chickens. Having the coop open for chickens would limit raccoons etc but allow late arrivals. Feeders for outdoor dogs….

…

On Mon, Jan 15, 2024 at 5:42 AM Eiritj ***@***.***> wrote: Thanks for the suggestion. I am already looking into cat doors that use RFID (DIY and ready products) since the cat is chipped, but they have several trade-offs. The major ones being range, integration with HA, and unreliable readings if not using batteries to power them. The easiest solution is a flap like the SureFlap combined with a door/windows sensor, but that leaves me with two more units that need new batteries every once in a while. Other than that RFID seems to be the way to go. I lack the skills required for image analysis. Perhaps a person could be hired to create what I need for image analysis, but even then there would be a question of support and updates, so it would not be feasible to outsource it. — Reply to this email directly, view it on GitHub <#1043 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALNOXUSRK3EYST4MBJ2EIYLYOUIY7AVCNFSM436JF6W2U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQMJTGIYTSNA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[kg5iru]

YES! I have 4 cameras covering the entirety of my family's chicken coop inside and out. I am interested in finding a way to keep up with which chickens are the most productive.

[rsteckler]

I'm having an issue where events from the custom models don't show up in the event list.
This "person" event showed up in the feed:

...but there's no associated "face" event.
face threshold is set at .7 and that event appears to have been .77

[ehn]

Got it! Thanks!

Yeah, detecting Amazon and other logos on packages would be neat.

In parts of the world where Amazon has outsourced logistics (which I think is most outside of the US), you won't be seeing Amazon logos on trucks or uniforms.

[ccutrer]

I always label UPS, FedEx, and Amazon when recognizable on a package (pretty rare for UPS and FedEx for me, but fairly common for Amazon).

[NickM-27]

yeah, in general I would recommend always labeling it if it is fully visible, may be useful in the future if it is expanded to more attributes

[interbiznw]

Amazon doesn't have any trucks of its own.

Except they do...

[NickM-27]

@interbiznw you missed

In my parts of the world

[mabed-fr]

Just to add that I am on my second model with more than 2000 labels the model becomes efficient on cars, people but finds cats and dogs almost everywhere and even in trees or bushes.

[mabed-fr]

Good ?

filters:
dog:
min_score: .7
threshold: .8
cat:
min_score: .65
threshold: .9
license_plate:
min_score: .6
person:
min_score: .65
threshold: .80
car:
min_score: .65
threshold: .85

[NickM-27]

Improvements to make labeling quicker and easier were pushed today, if you are planning to wait for automatic labeling then you will likely be waiting for quite a while

[mabed-fr]

I don't see what's wrong with #8229 ?

Sending the image to frigate+ with as many labels as possible is not that complicated?

Validation would still be mandatory

[NickM-27]

Sending the image to frigate+ with as many labels as possible is not that complicated?

first, to do that frigate would need to have a full copy of every object position for every frame (5 per second by default). that is way too much data to be kept for all events. the idea will be to rerun detection which does not exist currently in frigate.

second, frigate 0.13 is in RC so any changes will come in 0.14 which is under heavy development and likely won't have a beta for at least 3 months

[mabed-fr]

the idea will be to rerun detection which does not exist currently in frigate

Yes is the best solution !

I hope to find this in 0.14

[whatcom383]

I see you can detect FedEx, UPS, or Amazon... Why not USPS( UNITED STATES POSTAL SERVICE) They are the ones who deliver all my packages, i would like to detect those SPECIFIC vehicles

[ireuben]

Out of curiosity, what are people using the specific logo identification for? Do you have automation for specific delivery companies?

Personally I’d much rather have more general labels, particularly to start. Like an “animal” label, since I primarily care if animals (of any kind) are in places I don’t want animals to be (or at times I don’t expect them). Knowing if it’s a cat or a dog (and with less that 100% accuracy) isn’t nearly as useful to me as knowing if there is some kind of animal (cat, dog, mouse, possum, bear!) in my garage. Or some kind of package on my porch.

[NickM-27]

Frigate already supports taking the existing labels like cat, dog and you can change them to both be animal and it will work exactly in the way you're describing.

[ireuben]

That’s great that labels can be merged, but since there isn’t a more general “animal” label, there isn’t a way to either tag or recognize animals that aren’t cats or dogs. It seems like adding a new specific label for every type of animal would be a long and incomplete process. It’s like if instead of “vehicle” there were just “compact car” and “fedex van” labels, and over time we’d add support for “sports car”, “pickup truck”, “ups van”, “suv”, etc (and in the mean time those vehicles just wouldn’t be recognized at all, and marked as false positives if they were).
The way packages and people are recognized, with optional subtypes for a specific company’s logo, makes sense to me. It seems like animals could be something similar.
Btw, Apple’s HomeKit Video supports recognizing People, Animals, Vehicles, and Packages, which seems like a pretty useful starting point.

[NickM-27]

The way packages and people are recognized, with optional subtypes for a specific company’s logo, makes sense to me. It seems like animals could be something similar.

except that doesn't compare to animals. animals don't have logos, they are just different types of animals so the way that a car is detected as amazon would not be at all applicable to animals.

A dog and a bird are also entirely different shapes and I could see a much higher amount of false positives for users if there was just a generic animal label. Also, for me detecting a cat specifically is an important usecase and with animal I would lose that ability entirely meanwhile with the ability to merge each individual animal label back to animal functionality is not lost.

[blakeblackshear]

Merging could also potentially improve the scores. It may be hard to tell if something is a cat vs dog, but it can be damn sure its an animal. I'm not so sure that the same is true for bird vs horse for example. All this will require experimentation to find the right level of generalization.

[whatcom383]

Out of curiosity, what are people using the specific logo identification for? Do you have automation for specific delivery companies?

Personally I’d much rather have more general labels, particularly to start. Like an “animal” label, since I primarily care if animals (of any kind) are in places I don’t want animals to be (or at times I don’t expect them). Knowing if it’s a cat or a dog (and with less that 100% accuracy) isn’t nearly as useful to me as knowing if there is some kind of animal (cat, dog, mouse, possum, bear!) in my garage. Or some kind of package on my porch.

Personally I want to identify USPS, Vs UPS, Vs the neighbor driving down the street. I can't pick out the USPS delivery truck vs just a "vehicle" now.

[Caddyman68]

I see the option to subscribe for $50 a year is not available yet - is that correct?

[NickM-27]

It is available, you just need to request access by emailing. See #7932

[Caddyman68]

Thanks - I think I saw that some time back but just forgot when I saw this newer thread while searching for something else. I'll definitely subscribe, but want to collect the requisite number of images first.