Repository navigation
Frigate Performance Gains #24963
Replies: 1 comment 2 replies
|
Thanks for digging into this and for sharing the details. A lot of the diagram is already how Frigate works today. Frames are already written to a shared memory ring, camera processes already read frames in place without copying them, and only frame names (not pixels) are passed between processes. As of October, dev now keeps shm frames mapped instead of being reopened every frame (#24644), which removed most of the page faults and page zeroing. Also, OpenCV now runs single threaded in the camera processes (#24640), which removed a large number of thread context switches with many cameras. The custom ffmpeg muxer is the part that concerns me the most. Right now, Frigate won't overwrite a frame in the ring until it knows the rest of the pipeline is done with it. When processing falls behind, Frigate skips new frames rather than overwriting ones that are still in use. With ffmpeg writing directly into the ring, it keeps overwriting slots no matter how far behind processing is. Under heavy load (exactly when you'd want this to help), detection could run on a frame that is overwritten mid-detection, and the main process could find its frame already replaced, which means missing or corrupted snapshots and thumbnails. Sequence checks can detect this, but they can't prevent the frame from being lost. A custom muxer would also mean patching and building every ffmpeg version we ship across all of our image variants, and it wouldn't work for anyone using their own ffmpeg. The shared dispatch table is Hailo specific, and slots held across processes can leak if a detector process crashes or a chip hangs. Frigate already has an async detector runner (used by MemryX and Axelera) that keeps multiple requests in flight per device, so moving the Hailo detector to it should get you inference improvement with a much smaller change. What commit were your "before" numbers measured on? Could you also measure each change by itself on top of current dev, so we can see how much each one contributes? |
Uh oh!
There was an error while loading. Please reload this page.
Hi all
I've been using Frigate for the past year now. I started out with a desktop PC, using a GTX960 for decoding and an Advantech card with two Hailo8 chips for detection. I used to detect on the camera sub streams at 5fps, across 12 cameras. I then upgraded to a Dell PowerEdge R740 server with two Xeon Gold 6126 CPUs, moved the GTX960 over to still handle decode, and added a second Advantech card to bring my total Hailo8 count to 4. I also added another 6 cameras, so the total count is now 18.
Just for playing around, I decided to set the detection resolution to the native camera resolution (most around 2k). After doing this I would start skipping frames at around 270 detections/s, while my rough estimates were that I should only start skipping at about 450 detections/s. So I dug deeper and noticed that most of the time the Hailo chips were sitting idle, and the cause of the skipping was actually my CPUs limiting me. I was hitting pressure stall numbers in the high 50% region.
So I set up a test container in Proxmox and had Claude monitor it for a while, and this is what I found causing it: processed frames fell from 180/s to 28–70/s during busy periods, and recording segments were dropped. Profiling showed a flat profile, with about a third of all CPU time in the kernel. Most of that was copying frames through pipes, page faults and page zeroing, plus pipe-lock contention and up to 1.3M context switches/s.
I made a number of changes to the pipeline. The biggest thing is storing frames in shared memory and sharing them across as many processes as possible, because almost all that kernel time was spent moving frames around in memory.
Here is a little diagram of what the "redesigned" pipeline looks like. ● marks where a frame is kept, [COPY] is a full or partial copy or conversion, and (view) is zero-copy. Sizes assume a 2880×1620 yuv420p frame, about 6.7 MiB, and a 320×320 RGB tensor of 300 KiB.
For each detect frame, the only full-size copies are the two at the start: the GPU-to-RAM download and the copy into the ring. After that the camera, detector and main-process paths all read the same ring slot as views. The only further copies are small (the motion frame, about 300 KiB tensors), a per-object frame_cache copy, and some occasional ones: the once-a-second last-good frame, snapshots and MQTT jpegs, and previews, jsmpeg and birdseye in the Output process.
So with all cameras at native resolution and detection at 5fps, I would sit at an average CPU usage of about 40%, but with a huge amount of pressure stalling. After these changes I have all cameras at native resolution with detection at 10fps, average CPU usage is about 10%, and my max pressure stall across the last week is about 2%. So that is a huge jump in performance. Keeping more than one request in flight per Hailo also took each chip from 92 to 128 inferences/s.
I realize I'm pushing this system to process way more data than it was designed to do, but I feel like that performance gain would be a huge advantage to everyone. A couple of caveats: the shared memory ring needs a custom ffmpeg muxer, and the detection dispatch part is Hailo-specific for now. All of this was done on 0.19 dev.
So asking the devs: would you guys be interested in me creating PRs for these changes? I'll make them in small incremental bits so that they can be properly reviewed and then eventually adopted.
All reactions