Changing Attempts Timing/Delay #83

Enclavet · 2021-08-31T16:53:16Z

Curious about the way double-take is grabbing snapshots/latest files from the API. A few things I do want to mention:

When trying to grab latest files and grabbing snapshots from MQTT, double-take does this in parallel. What happens is that frigate will use a snapshot that might be older than the match found from the latest.jpg processing thread. Meaning if there are a multiple cameras, and double-take is used for presence detection, I could potentially be matched from the snapshot thread in a place I am no longer in. I decided to turn off the MQTT snapshot process for this reason.

The other thing I noticed is the attempts value, I notice that when grabbing latest images from frigate, that it does so in a sequential way without delay/timings. From the logs double-take makes 20 attempts in 1-2 seconds and doesn't find a match. But in the frigate event, my face might not show up until 3-4 seconds after the event starts, missing my face.

This has to do with FPS as I am using 5 FPS on the camera, if the attempts try to grab 20 frames in 1-2 seconds it will be analyzing the same images over and over.

Wondering if we can add a delay between attempts for this reason.

I also noticed is that switching to a slower model actually helps this as the delays between attempts is greater.

jakowenko · 2021-09-01T16:46:20Z

Hey there, thanks for checking out the project and the detailed issue. Let me try to explain some of it as well and make sure we are on the same page. The processing of the Frigate images / MQTT image is a little confusing and the MQTT one is done separately then the event driven one.

Frigate publishes a message to `frigate/events`:

Double Take then begins to process this event using images from Frigate's API. These images are /api/${camera}/latest.jpg and /api/events/${id}/snapshot.jpg. The snapshot image in the double-take:latest is set to 0 currently due to a bug on the Frigate API side. If you're running Frigate 0.9.0 then you should be fine to increase this. The big difference with the snapshot image here is that it's based off the event ID. This is different than the MQTT image described below.

These two images are processed in parallel with the configured detectors. You're right there is no delay between processing events, but the time to process an image should add delay between each loop. The default amount of retries is 10 per image. Each loop should pull in the current frame from those two images and then pass it for detection.

Are you getting any errors in the 20 attempts in 1-2 seconds? The detector you're using is responding fast enough to process 20 images? I see that the slower model helps with the delays, so your machine must process them quicker than mine which is why I haven't seen the quicker API requests. I can add in a delay to help with this though.

Frigate publishes a message to `frigate/+/person/snapshot`:

Frigate publishes a JPEG encoded image to this topic when certain conditions are met on the Frigate side. There is no context around this image other than the camera name and JPEG frame. Double Take uses this image to process it through a single recognition loop. It's only processed one time unless Frigate publishes a new image to that topic.

You're right, if you try to use this image for presence detection then you could have reports you are in a room after you left it because Frigate published the best snapshot when the event was done.

I run into this issue right now as well with my setup, but I'm not sure the best way to overcome it. I orginally didn't process the frigate/+/person/snapshot because the snapshot.jpg image was better since it's tied to the ongoing Frigate event. I only added support for the MQTT image when I discovered Frigate had a bug with the snapshot.jpg image, but now that it's fixed in 0.9.0, maybe I just make the processing of the MQTT image a config setting on the Double Take side.

Sorry for the long response, let me know if this makes sense.

Enclavet · 2021-09-01T18:58:57Z

Your response is spot on.

Processing Attempts
Nope no errors. Average detection for the Compreface with 1080p images is 100-150ms using the MobileNet model.

Having a tunable delay would probably be helpful for most people and the easiest to implement. I might just compile your code with my delay hardcoded to test it out. Basically I need to put a delay that will take into account the average processing time of my detector (in my case 100-150ms) + the time a new frame might be recorded (5fps would be every 200ms).

A better way probably to calculate the delay depending on the fps of the source. So say double-take finds that a detection is 100ms and it knows that a frame will only come every 200ms, it would delay by 100ms dynamically.

The best way to solve this is to put a face detection model co-compiled with the object detection model that frigate uses, to have a new object detection event (Face detection vs Person detection). I might do this with a custom model.

MQTT Snapshots
I ended up just turning off MQTT snapshots from the Frigate side. So having a way to turn this off in Double-take would be helpful.

jakowenko · 2021-09-02T04:48:30Z

I pushed a couple updates to the beta build to help address these issues. Let me know what you think.

frigate:
  url: https://frigate-url.com
  attempts:
    mqtt: false # false: do not process mqtt snapshot topic images | true by default
    delay: .25 # add a delay expressed in seconds between each detection loop

Enclavet · 2021-09-02T19:05:10Z

I installed this in the morning and have been testing it in the morning. I set it to 100ms delay + increasing the attempts to 40 tries (I know probably taxing my system). I see that my detection is around 4-5 frames per second for 9-10 seconds which pretty much covers all/most frames in the event.

Thanks for the change.

ozett · 2021-09-03T08:19:57Z

What about making a Feature-Request to Frigate to send .jpg-image with every /MQTT/event/ ?

(Also for some other reasons:
This way one could benefit from the event-context (also for other purposes) and get images over single channel without pulling them.)

Enclavet · 2021-09-03T17:45:05Z

Frigate already sends a "best" snapshot with the MQTT/event/.

The problem is Frigate triggers an event based on a "Person" object detection because their models only do object detection. It would be better if they implement a face detection model to have a Face event that could be sent to Double-Take.

ozett · 2021-09-03T18:36:59Z

Frigate already sends a "best" snapshot with the MQTT/event/.

but it seems the .jpg is missing in the mqtt/event payload...
blakeblackshear/frigate#1560

fully agree:

The problem is Frigate triggers an event based on a "Person" object detection because their models only do object detection. It would be better if they implement a face detection model to have a Face event that could be sent to Double-Take.

or a way to feed custom models from motion-trigger?

ozett · 2021-09-05T11:09:21Z

New API event/endpoint: clips!
blakeblackshear/frigate#1502 (reply in thread)

There is a new endpoint not yet in the http docs: /api/events/<event_id>/clip.mp4. This will use ffmpeg to dynamically create an mp4 in the tmp directory from the recording segments and then tell nginx to return that file as the result. It is shockingly fas

# [0.11.0](v0.10.2...v0.11.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437))

# [1.0.0](v0.10.2...v1.0.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### chore * release ([3dddd40](3dddd40)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437)) ### BREAKING CHANGES * v1.0.0

# [1.0.0](v0.10.2...v1.0.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### chore * release ([824db79](824db79)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437)) ### BREAKING CHANGES * v1.0.0

jakowenko · 2021-09-21T19:22:31Z

Closing since this is now included in the 1.0.0 release.

jakowenko added a commit that referenced this issue Sep 2, 2021

feat: enable or disable frigate mqtt topic snapshot processing (#83)

3bf2bea

jakowenko added a commit that referenced this issue Sep 2, 2021

feat: add a delay expressed in seconds between each detection loop (#83)

23dc29e

jakowenko closed this as completed Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changing Attempts Timing/Delay #83

Changing Attempts Timing/Delay #83

Enclavet commented Aug 31, 2021 •

edited

jakowenko commented Sep 1, 2021

Enclavet commented Sep 1, 2021 •

edited

jakowenko commented Sep 2, 2021

Enclavet commented Sep 2, 2021

ozett commented Sep 3, 2021

Enclavet commented Sep 3, 2021

ozett commented Sep 3, 2021

ozett commented Sep 5, 2021

jakowenko commented Sep 21, 2021

Changing Attempts Timing/Delay #83

Changing Attempts Timing/Delay #83

Comments

Enclavet commented Aug 31, 2021 • edited

jakowenko commented Sep 1, 2021

Frigate publishes a message to frigate/events:

Frigate publishes a message to frigate/+/person/snapshot:

Enclavet commented Sep 1, 2021 • edited

jakowenko commented Sep 2, 2021

Enclavet commented Sep 2, 2021

ozett commented Sep 3, 2021

Enclavet commented Sep 3, 2021

ozett commented Sep 3, 2021

ozett commented Sep 5, 2021

jakowenko commented Sep 21, 2021

Enclavet commented Aug 31, 2021 •

edited

Frigate publishes a message to `frigate/events`:

Frigate publishes a message to `frigate/+/person/snapshot`:

Enclavet commented Sep 1, 2021 •

edited