Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Add Zone support from Frigate #31

Closed
wills106 opened this issue Apr 15, 2021 · 24 comments · Fixed by #43
Closed

[FR] Add Zone support from Frigate #31

wills106 opened this issue Apr 15, 2021 · 24 comments · Fixed by #43
Labels
enhancement New feature or request

Comments

@wills106
Copy link

wills106 commented Apr 15, 2021

Would it be possible to only pass images to Deepstack when the motion from Frigate is within a Zone?

Use Case:

I have a camera looking down my drive.
I have a Motion mask on the road as I am not bothered about cars / people in the road.
I then have a zone this side (drive) of the footpath / sidewalk.

I have a zone defined as:
drive_zone_0
under "drive" camera

I am not bothered / interested in Face Detection of people walking up and down the street. But I would like to run Face Detection on people who walk down my drive towards the house. ( drive_zone_0 )

@jakowenko jakowenko added the enhancement New feature or request label Apr 16, 2021
@jakowenko
Copy link
Owner

Hey! Definitely seems possible, I could add another environment variable you could pass in to specify which cameras and zones you want to watch. Maybe something like FRIGATE_ZONES: camera-name:zone-name, drive:drive_zone_0?

@wills106
Copy link
Author

Yea that sounds an easy enough environment to specify.

That way when Frigate detects a person entering the zone, the image gets sent to Deepstack.
Also by this point the person would be facing straight into the camera, so would result in a better image quality / detection. As it wouldn't be passing a side on image, which can happen if it responds just before the person turns into the drive.

@srvoets
Copy link

srvoets commented Apr 17, 2021

Hey! Definitely seems possible, I could add another environment variable you could pass in to specify which cameras and zones you want to watch. Maybe something like FRIGATE_ZONES: camera-name:zone-name, drive:drive_zone_0?

exactly, Id like to set up a notification/TTS for when a match is found for my frigate zone 'front_porch'

jakowenko added a commit that referenced this issue Apr 20, 2021
Publish matches by camera name and add support to specify Frigate zones - #31
@jakowenko
Copy link
Owner

@wills106 @deputyd0ng, I pushed a new beta build to include the changes we talked about. The README shows the new options that can be passed to the container.

Match results are also now published to MQTT in this format: double-take/cameras/:camera-name. Did you have any opinions on how you want the values stored for that topic? Right now it's an array of objects for each person found in the image.

Let me know what you think of these changes and if there's anything you'd like to see changed.

Screen Shot 2021-04-20 at 11 41 26 AM

@wills106
Copy link
Author

Just wanted to let you know that configuration with zones is working fine.
As it's putting less load on Frigate only looking at zones I am finding Frigate doesn't crash so much.

Not done anything with MQTT yet, but the topics look ok and make sense.

@srvoets
Copy link

srvoets commented Apr 27, 2021

@wills106 @deputyd0ng, I pushed a new beta build to include the changes we talked about. The README shows the new options that can be passed to the container.

Match results are also now published to MQTT in this format: double-take/cameras/:camera-name. Did you have any opinions on how you want the values stored for that topic? Right now it's an array of objects for each person found in the image.

Let me know what you think of these changes and if there's anything you'd like to see changed.

Screen Shot 2021-04-20 at 11 41 26 AM

Working on this now. Im trying to make a node-red entity, similar to the example that was created, I have the node just called "front door face"

I am using essentially the same node from the example, but ti doesnt seem to be working, what would be the best way to parse out any match from double-take/cameras/front_door topic? I thought just using the parsed JSON object would work, but it aint.

And thanks again! If I was more competent with this stuff Im sure It would be even more useful for me.

@jakowenko
Copy link
Owner

@deputyd0ng does your node look something like this? The output of this should be an array of objects, each one represents a matched person. Can you post a screenshot of what your entire Node-Red flow looks like? I can try to help debug.

Screen Shot 2021-04-27 at 1 11 00 AM

@srvoets
Copy link

srvoets commented Apr 28, 2021

image

Do I need to set my QOS to 0 (no idea what that is)

This is my second node, basically the same as the example:

image

@jakowenko
Copy link
Owner

@deputyd0ng Yes, try to set the QOS to 0. I'd also put a debug node after the topic to make sure you're getting the data. The data for the camera topic is an array, the entity node you have above I believe will only work if you subscribe to just the notifications for a specific person. If you only care about the first face of the results you can probably just replace payload with payload[0] (not 100% sure if this will work, but I don't see why not).

I dropped payload.room from the current beta build since it was just a manipulated string from the camera string. Since this is a front door HA entity, you'll probably want to set the state of this node to be payload[0].match.name so HA will show the most recent face it found.

@jakowenko jakowenko mentioned this issue Apr 29, 2021
@jakowenko
Copy link
Owner

@deputyd0ng I added better documentation and some samples for HA + Node-Red

@mannp
Copy link

mannp commented May 14, 2021

Hi @jakowenko, i am using the beta docker build and a config.yml file trying to configure multiple zones per frigate camera....the docs show the following format;

  zones:
    - camera: driveway
      name: zone-1

This for me results in -> zones: [ [Object] ]

Adding this way;

  zones:
    - homeparkingfull0:zone_drive

Results in -> zones: [ 'homeparkingfull0:zone_drive' ] in the log :)

Adding;

  zones:
    - camera1:zone1
    - camera1:zone2
    - camera1:zone3
    - camera1:zone4
    - camera2:zone1
    - camera2:zone2

Appears to populate the zones correctly in the log, but goes against the docs.

I wondered what the correct way (if supported) is of stipulating multiples zones per camera.

Thanks

@jakowenko
Copy link
Owner

jakowenko commented May 14, 2021

Hey @mannp, the docs should be correct even though the logs show Object. I'll work on correcting this so it's readable. Maybe I'll just move it to a page on the UI. The issue right now is that it's too nested for the logs to show the values. But if you follow the docs it should work.

Let me know if you are still running into issues and I'll let you know when I correct the output to show the value.

@mannp
Copy link

mannp commented May 14, 2021

Thanks for that @jakowenko :)

So multiple zones per camera is multiple name: zone.x lines under the camera definition?

@jakowenko
Copy link
Owner

jakowenko commented May 15, 2021

Multiple zones for the same camera should look like this. I'm on mobile, sorry if formatting is off.

zones:
  - camera: driveway
    name: zone-1
  - camera: driveway
    name: zone-2

@mannp
Copy link

mannp commented May 15, 2021

Hi @jakowenko I have that setup now, although nothing showing in doubletake as yet.

It seems frigate crashes alot with doubletake for some reason.

Maybe I'll just move it to a page on the UI.

Level of work permitting this would be great, for a noob, it is hard to tell if doubletake has what it needs to work and all connections are okay etc. There are no errors in the log, but nothing showing up in doubletakes gui right now.

Had a few frigate events but nothing showing in the doubletake log that the event is being processed....will give it some more time to see if things imrpove.

@jakowenko
Copy link
Owner

@mannp can you post your config and I can see if everything looks right.

There does seem to be some issues around Frigate snapshot images. There were some open issues on Frigate's repo around connections not being closed. You should try setting frigate.attempts.snapshot to 0 to see if that resolves your issue. I've been running my current setup like that for 2 weeks now without any Frigate crashes.

You can also hit the API directly to force an image to show up in the GUI to make sure everything seems like it's working. I have it documented here.

@mannp
Copy link

mannp commented May 16, 2021

@jakowenko this is my current config, but getting errors in the log around the zone definition, see below.

Thanks for any insight as to my issues 👍🏻

See last log where doubletake is picking up camera definitions as zones?

Edit: Ignore the yi-1 duplicate in the first log, as that was my error and should have been hv-1

- camera: yi-1
      name: zone_keepclear
frigate:
  url: http://192.168.30.60:5000
  zones:
    - camera: homeparkingfull0
      name: zone_drive
    - camera: yi-1
      name: zone_keepclear
    - camera: yi-1
      name: zone_roaddoor
    - camera: yi-1
      name: zone_keepclear
    - camera: hv-1
      name: zone_drive
    - camera: hv-2
      name: zone_ourpath  

Initial log with deepstack enabled and my error in the zone definition

deepstack process error: Cannot read property 'map' of undefined
latest attempt 2
1621161143.562615-gh3zbl - homeparkingfull0 zone not on approved list
deepstack process error: Cannot read property 'map' of undefined
1621161168.762443-pak4xe - homeparkingfull0 zone not on approved list
1621161170.911035-l1bq1g - yi-1 zone not on approved list
1621161185.484552-ctckc4 - yi-1 zone not on approved list
latest attempt 3
deepstack process error: Cannot read property 'map' of undefined
1621161178.197109-rmzndh - homeparkingfull0 zone not on approved list
latest attempt 4
deepstack process error: Cannot read property 'map' of undefined
1621161170.887676-5pfwgv - still processing previous request
1621161156.707334-6eitm0 - still processing previous request
latest attempt 5
deepstack process error: Cannot read property 'map' of undefined
1621161171.34417-ptct30 - homeparkingfull0 zone not on approved list
latest attempt 6
deepstack process error: Cannot read property 'map' of undefined
1621161171.34417-ptct30 - homeparkingfull0 zone not on approved list
latest attempt 7
deepstack process error: Cannot read property 'map' of undefined
1621161143.562615-gh3zbl - homeparkingfull0 zone not on approved list
latest attempt 8
deepstack process error: Cannot read property 'map' of undefined
1621161178.197109-rmzndh - homeparkingfull0 zone not on approved list
1621161185.484552-ctckc4 - yi-1 zone not on approved list
latest attempt 9
deepstack process error: Cannot read property 'map' of undefined
latest attempt 10
deepstack process error: Cannot read property 'map' of undefined

:
{
id: '1621161170.887676-5pfwgv',
duration: 8.79,
timestamp: '5/16/2021, 11:33:12 AM',
attempts: 10,
camera: 'hv-1',
zones: [ 'zone_drive' ],
matches: []
}
5/16/2021, 11:33:12 AM
done processing hv-1: 1621161170.887676-5pfwgv in 8.79 sec
1621161185.484552-ctckc4 - yi-1 zone not on approved list
1621161170.911035-l1bq1g - yi-1 zone not on approved list
1621161206.354517-r26dt5 - homeparkingfull0 zone not on approved list
1621161206.354517-r26dt5 - homeparkingfull0 zone not on approved list
1621161206.354517-r26dt5 - homeparkingfull0 zone not on approved list

Second log with deepstack disabled and the zone definition corrected...I am still seeing zone errors though, will check them again...


done processing hv-1: 1621161821.151975-1kwf5m in 7.93 sec
1621161815.965575-0ip2p7 - homeparkingfull0 zone not on approved list
1621161821.151975-1kwf5m - hv-1 zone not on approved list
1621161823.716671-vp36dh - skip processing on end events
1621162003.636007-mmy11r - label not a person, car found
1621162004.008706-672sla - label not a person, car found
1621162004.110347-og5zjv - yi-1 zone not on approved list
1621162005.962307-01vxsw - homeparkingfull0 zone not on approved list
1621162006.346672-mccnyv - homeparkingfull0 zone not on approved list
1621162004.008706-672sla - label not a person, car found
1621162004.110347-og5zjv - yi-1 zone not on approved list
1621162005.962307-01vxsw - homeparkingfull0 zone not on approved list
1621162005.962307-01vxsw - homeparkingfull0 zone not on approved list
1621162004.110347-og5zjv - yi-1 zone not on approved list
1621162004.008706-672sla - skip processing on end events
1621162006.346672-mccnyv - homeparkingfull0 zone not on approved list
1621162003.636007-mmy11r - skip processing on end events
1621162006.346672-mccnyv - homeparkingfull0 zone not on approved list
1621162071.531892-ae0hy4 - yi-1 zone not on approved list
1621162071.531892-ae0hy4 - yi-1 zone not on approved list
1621162084.350896-uo6gi9 - homeparkingfull0 zone not on approved list
1621162084.350896-uo6gi9 - homeparkingfull0 zone not on approved list

@jakowenko
Copy link
Owner

jakowenko commented May 17, 2021

@mannp the config looks ok. I did just push a new beta build that includes a config route you can hit via the API until I add it to the UI. I also fixed the logs to show the full config object.

In terms of the logs you posted, it appears DeepStack is having issues, or is not sending back the expected response. It's related to this error, deepstack process error: Cannot read property 'map' of undefined. Try restarting DeepStack and see if that fixes it.

Try hitting the API manually to see if you can get an image to show up on the UI.

curl -X GET "http://localhost:3000/api/recognize?url=https://jakowenko.com/img/david.92f395c6.jpg" \
-H "Content-type: application/json"

I'm also curious what you think about the config.yml setup for the cameras / zones. I wonder if it can be improved and be more straightforward with something like this. In the below example, only Frigate events would be processed if they came from the backyard in zone deck or yard and any events from the driveway camera. Having this in two objects right now may be a bit more confusing and I like the idea of trying to combine them.

frigate:
  url: http://192.168.1.2:4000
  cameras:
    - name: backyard
      zones:
        - deck
        - yard
    - name: driveway

Same idea, slightly different format.

frigate:
  url: http://192.168.1.2:4000
  cameras:
    backyard:
      zones:
        - deck
        - yard
      driveway:

I think either of these are better than what I currently have. What do you think?

@mannp
Copy link

mannp commented May 17, 2021

@jakowenko the new beta worked better for me, with the same config I got pictures and attempts at matches in the doubletake gui.... some car hub caps recognised as faces, but at least some progress thanks :)

Edit: After a full day of activity in the log I have no more activity in the gui....seems hit and miss for me, not sure why or how to fix it... :-/

My log now shows the following;

zones: [
{ camera: 'homeparkingfull0', name: 'zone_drive' },
{ camera: 'yi-1', name: 'zone_keepclear' },
{ camera: 'yi-1', name: 'zone_roaddoor' },
{ camera: 'hv-1', name: 'zone_keepclear' },
{ camera: 'hv-1', name: 'zone_drive' },
{ camera: 'hv-2', name: 'zone_ourpath' }

Personally the log would make more sense if 'name' was replaced by 'zone' and so I am leaning more towards the latter config format, just keeping things simple (obviously 'driveway' needs to be indented to the left one, just a github formatting think i guess).

Happy to create another issue, rather than post here, but re deepstack, its log is as follows;

[GIN] 2021/05/17 - 10:29:51 | 403 | 29.025µs | 192.168.30.69 | POST /v1/vision/face/recognize
[GIN] 2021/05/17 - 10:29:53 | 403 | 33.722µs | 192.168.30.69 | POST /v1/vision/face/recognize

The IP is correct for an api call from doubletake, so no issues there, but perhaps newer versions of deepstack return slightly different responses or the api call changed? Isn't there supposed to be a file url in the POST call..

DeepStack: Version 2021.02.1

Also the new config api call also works for me :)

@mannp
Copy link

mannp commented May 17, 2021

Further testing and using a previous face image from frigate, the logs seem better but still nothing further in the gui.

curl -X GET "http://192.168.30.69:3000/api/recognize?url=http://192.168.30.60:5000/clips/homeparkingfull0-1621245633.860043-hwr6z9.jpg&results=all"

MQTT: connected
MQTT: subscribed to frigate/events
5/17/2021, 5:05:28 PM
processing double-take: 51c9f669-3aa8-4c4b-bdff-68f3a49cf19e
deepstack process error: Cannot read property 'map' of undefined
5/17/2021, 5:05:28 PM
processing double-take: 51c9f669-3aa8-4c4b-bdff-68f3a49cf19e
deepstack process error: Cannot read property 'map' of undefined

:
{
id: '51c9f669-3aa8-4c4b-bdff-68f3a49cf19e',
duration: 2.26,
timestamp: '5/17/2021, 5:05:30 PM',
attempts: 1,
camera: 'double-take',
zones: [],
matches: [],
results: [
{
duration: 2.26,
type: 'manual',
attempts: 1,
results: [
detector: 'deepstack',
duration: 0,
attempt: 1,
results: [],
filename: 'babc2ebd-b3e8-4261-8bc8-c16517975d76.jpg'
},
detector: 'facebox',
duration: 2.25,
attempt: 1,
results: [],
filename: 'babc2ebd-b3e8-4261-8bc8-c16517975d76.jpg'
]
}
]
}
5/17/2021, 5:05:30 PM
done processing double-take: 51c9f669-3aa8-4c4b-bdff-68f3a49cf19e in 2.26 sec
{
id: '51c9f669-3aa8-4c4b-bdff-68f3a49cf19e',
duration: 2.26,
timestamp: '5/17/2021, 5:05:30 PM',
attempts: 1,
camera: 'double-take',
zones: [],
matches: [],
results: [
{
duration: 2.26,
type: 'manual',
attempts: 1,
results: [
detector: 'deepstack',
duration: 0,
attempt: 1,
results: [],
filename: 'babc2ebd-b3e8-4261-8bc8-c16517975d76.jpg'
},
detector: 'facebox',
duration: 2.25,
attempt: 1,
results: [],
filename: 'babc2ebd-b3e8-4261-8bc8-c16517975d76.jpg'

@jakowenko
Copy link
Owner

@mannp you do have vision face and detection setup for your DeepStack container, right? It should look something like this.

  deepstack:
    container_name: deepstack
    image: deepquestai/deepstack
    restart: unless-stopped
    environment:
      VISION-FACE: 'True'
      VISION-DETECTION: 'True'

@mannp
Copy link

mannp commented May 17, 2021

@jakowenko I am not using any env variables, since I have only starting setting up with the config.yml file using the beta.

Oops for deepstack, let me check...

Edit: So that explains the deepstack issue, as I now have this is the deepstack log

Edit: Facepalm .. apologies for that noise...

DeepStack: Version 2021.02.1
/v1/vision/face
/v1/vision/face/recognize
/v1/vision/face/register
/v1/vision/face/match
/v1/vision/face/list
/v1/vision/face/delete
/v1/vision/detection
v1/restore
[GIN] 2021/05/17 - 17:55:47 | 500 | 1m0s | 192.168.30.69 | POST /v1/vision/face/recognize

The manual recognize doesn't get added to the gui though.

Not sure what has changed since the flurry of images in the gui first thing today...

@jakowenko
Copy link
Owner

jakowenko commented May 17, 2021

Did the deepstack process error: Cannot read property 'map' of undefined go away once you rebuild the DeepStack container with the env vars? We should be getting close now that DeepStack is setup correctly.

Feel free to message me on Discord if that's easier, my name should be Jako#4153.

@mannp
Copy link

mannp commented May 17, 2021

Did the deepstack process error: Cannot read property 'map' of undefined go away once you rebuild the DeepStack container with the env vars? We should be getting close now that DeepStack is setup correctly.

No the error hasn't gone away unfortunately :) but I haven't had it enabled for 99% of the testing above, so its really a nice to have for me.

I have been using facebox most of the time alone.

Feel free to message me on Discord if that's easier, my name should be Jako#4153.

Done and thanks :)

@jakowenko jakowenko mentioned this issue May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants