[Question]: Why is frigate choosing worst frame for snapshot?

[rsteckler]

What is your question:

Frigate often chooses a really bad frame for the snapshot, which makes uploading to frigate+ not very valuable. Here's an example:
The snapshot for the event is this.

While it's true there's a person there, these were some previous frames that could have been chosen:

This happens often, which isn't helpful when the snapshot frame gets sent to my messages, and not very helpful for uploading to Frigate+ to help the models.

Is there a way to change which frame is chosen?

[NickM-27]

By default frigate uses the score and the size of the object to determine which snapshot to keep. It also is bound by best image timeout so it can only save a best image once every 60 seconds by default. That might be part of the issue here, that the best snapshots in the middle can't be considered because of the best snapshot.

If you use frigate+ then an additional factor is considered which is that frigate will choose the image with the best face (person) or the best license plate (car)

[rsteckler]

First, I'm always impressed by how responsive you are to questions.

I'd love to find out why this happens regularly. If there's anything I can do to debug it, I'd really like to figure it out.
Here's the first detection frame:

Here's when the person entered a zone:

And here's the exit:

The whole event was only 15 seconds, so I don't think a 60 second timeout would matter.

The chosen snapshot frame can't possibly be the best frame/face frame because the confidence is only 67%, which is below the detection value, so it must be lower than other frames. It also has no face detected, where I assume other frames would, given the images. It could be the largest area, but that seems like an odd criteria to choose. I do use Frigate+.

Let me know if there's anything I can do to find out why it keeps choosing frames like this, or how I can tune it to choose better ones!
Thanks!

[NickM-27]

The whole event was only 15 seconds, so I don't think a 60 second timeout would matter.

It would because that means only one best snapshot can be considered for the lifetime of the event

It might be helpful to see your config

It could be the largest area, but that seems like an odd criteria to choose.

Not really, an object might score well but be small and that's not helpful. An object being larger means there will be more detail and more recognizability

[rsteckler]

Took a look at the 'is_better' method here:

frigate/frigate/object_processing.py

Line 69 in af320c8

def is_better_thumbnail(label, current_thumb, new_obj, frame_shape) -> bool:

I think that explains it and I think it can be improved. Right now, it chooses the latest frame for the snapshot if the size of the object is 10% bigger than the previous best frame (unless the old best frame had a face and this one doesn't). I think that's why I'm getting so many of these. I have a few cameras setup where people/cars become obstructed the closer they get:

My suggestion would be tunable parameters for these values. Something like:

best_frame_tunables:
  size_delta: 1.1
  quality_delta: 0 #any improvement chooses the new frame
  quality_drop_allowed_on_size_delta: .05 #only allow a 5% drop in quality when considering updating the best frame based on size

In addition to the above driveway camera, here my other two camera setups where objects are obscured as they get larger and I see consistently obscured snapshots.

Front door - People are huge right at the bottom of the frame, but it's a top-down view that loses any detail of their body/features/cothing.

Front Gate - Cars on the road come closest right as they go behind the bushes:

Here's the config for that camera, but I don't have anything interesting in there, I don't think.

camera_driveway:
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 1080
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
      - path: rtsp://localhost:8554/camera_driveway
        input_args: preset-rtsp-restream
        roles:
        - audio
        - detect
    audio:
      enabled: true
    detect:
      width: 1920
      height: 1080
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      crop: false
      height: 1080
      retain:
        default: 365
        objects:
          person: 50
    motion:
      mask:
      - 1300,30,1850,30,1850,100,1300,100
    zones:
      driveway_notify:
        coordinates: 
          160,526,327,666,141,1080,1920,1080,1920,0,1009,0,1006,204,545,271,438,229,276,249
    objects:
      filters:
        car:
          mask:
          - 1625,430,1625,275
        person:
          mask:
          - 146,626,234,629,236,544,146,539
          - 146,626,234,629
      track:
      - person
      - dog
      - cat
      - car
      - face
      - amazon
      - fedex
      - ups
      - license_plate

[NickM-27]

to be clear there is a feature request for making the logic better #5256

I think tunable config for this would be a last resort. We have plenty of signals that can be listened to. But in any case I am not sure how important this is given that the images used in the ui for the next version of frigate won't use this logic at all

[rsteckler]

Got it. Sounds like this is already a topic of discussion, so I'll close this and link it to the other thread.

Thanks!