Calculating the peak number of detections

[tgudlek]

Hey!

I'm new to Frigate, but have been loving it so far. Thank you for what you've built so far! :)

Based on my current understanding, the detection happens after a movement has been detected. A bounding box is then placed around the movement, and the bounded pixels are sent for detection.

My question is -- what happens if the bounding box is larger than the WxH that the model is designed to take as input? Is the content simply downscaled to fit the window (say, 320x320), or are multiple detection issued in order to tile the full movement space?

I'm asking this because I want to understand what kind of load I can expect when there's a large movement happening in front of the camera. I'm trying to get the license plate recognition to work through Frigate with my Aqara G5 Pro camera which offers 1520p, 1080p, 720p, and 360p on top of RPi5+Hailo8L. I'm trying to get it to work using the 720p stream for detect, and am considering the 1080p stream as well as bumping the model to yolo8n. However, both decisions are connected to the peak detection count which I don't know how to compute since I don't know how large movements are handled.

[hawkeye217]

While this is primarily talking about masks, this explanation in the official documentation will probably be helpful: https://docs.frigate.video/configuration/masks/#further-clarification

"Regions" are the green boxes you see in the debug view. These are the regions of the image resized down to the model's input and run through the object detector.

Another helpful page in the official docs would be this one on choosing a detect resolution: https://docs.frigate.video/frigate/camera_setup/#choosing-a-detect-resolution

[tgudlek]

Ah, perfect, the second link is just what I was looking to know. Thank you!

So, to summarize --

Using a larger stream for detection would put extra load on my CPU through decoding (because RPi5 doesn't support hwaccl for H264, and that's the only thing my cameras support), and there's a small additional overhead for downscaling the region to model's input dimensions.

Using a larger stream for detection would improve detection of smaller objects because there are more pixels to be sent to the detector, but won't do anything for larger objects since they're getting downscaled anyway.

The maximum number of detections per frame is determined by the largest number of disjoint movement zones, for which the upper bound then is equal to how many tiles of WxH, where WxH is model input dimensions, are needed to tile the full frame (not counting the region-connecting logic). Howver, the actual maximum number of detections is very much determined by how aggressive the region-connecting logic is.