[Bug]: 0.18 TrackedObjectProcessor copies every detect frame on one thread (CameraState.update np.copy), silently capping aggregate detect resolution

[jvallery]

Describe the problem you are having

TrackedObjectProcessor (the detected_frames_processor thread) does a full np.copy of every camera's detect frame on every frame. Because that work is serialized on one thread, it caps the aggregate detect resolution a single Frigate instance can process. Processed fps falls below captured fps and frames are dropped silently: skipped_fps stays at 0, with nothing logged, while the host is mostly idle.

• Setup: 40 cameras, about 200 detect fps in aggregate, detect resolutions mixing 4K, 1440p, 1080p and D1, all at 5 fps.
• Symptom: with about 18 detect streams at 4K class, processed fps was 127–190 of 201 captured. The node was at least 40% idle, and the detectors (4 ONNX/TensorRT instances at 17–22 ms) and ffmpeg kept up.
• Workaround that confirmed the cause: moving four cameras to 2560x1440 and two to D1 restored processed = captured, with the thread at about 75% busy.

Profile: a 60 s py-spy sample of the main process shows detected_frames_processor about 97% busy:

• 49% in np.copy of the full detect frame on every frame: frigate/camera/state.py, CameraState.update(), self._current_frame = np.copy(current_frame). It is line 613 on master and line 604 on dev, so both are still affected.
• About 12% in per-frame shared-memory map and unmap (frame_manager.get / close).
• Snapshot writes: TrackedObject.write_snapshot_to_disk() encodes the clean full-resolution WebP synchronously on this thread, taking 1.0–1.9 s per 4K object end. During that time the frame loop stalls for every camera; the detected-frames queue holds about 0.4 s at 200 fps. In our data, 64.7% of object ends on cameras of 8 MP or more were followed by a fleet-wide drop burst within 4 s, against a 4.4% chance rate.

Microbenchmark (same container, separate process, one I420 frame per iteration):

Detect size	np.copy	np.copyto into a preallocated buffer
3840x2160	4.2–5.5 ms (up to 7.9 ms on a busy vCPU)	5.0 ms
2560x1440	1.4 ms	2.1 ms
1920x1080	0.6 ms	0.9 ms
704x480	0.03 ms	0.03 ms

Preallocation does not help, and there are no page faults: the copy is memory-bandwidth bound. The resulting cost model is ≈2.35 ms per frame plus ≈6 ms per 4K frame, so one thread bounds the aggregate detect pixel rate regardless of available cores.

Suggestions (any one helps; together they remove the ceiling):

1. Copy the current frame lazily.
   • _current_frame is only read on demand: API latest, MQTT camera images, chat and VLM jobs, debug replay.
   • Instead, keep the latest shared-memory frame mapped, holding its slot until the next frame, which is already how previous_frame_id works. Copy only inside get_current_frame().
   • This needs the capture ring to leave the held slot unwritten, for example with one spare slot or a reference count.
2. Move snapshot and thumbnail encoding off the frame loop. Run write_snapshot_to_disk() and write_thumbnail_to_disk() on a worker thread or executor; the frame_cache entry is already a private copy.
3. Update camera states in parallel. Each CameraState is independent, and NumPy releases the GIL for large copies, so a small thread pool keyed by camera would use the idle cores.

Happy to test a build or branch against this workload and report processed versus captured fps.

Steps to reproduce

1. Run many cameras with high-resolution detect, for example 15–20 cameras at 3840x2160 with detect.fps: 5, alongside lower-resolution ones.
2. Compare sum(camera_fps) with sum(process_fps) in /api/stats while skipped_fps stays at 0.
3. py-spy dump or py-spy record the main Frigate process and look at the detected_frames_processor thread: most of its time is in CameraState.update → np.copy.
4. Lower some cameras' detect resolution: processed fps recovers in proportion to the pixels removed.

Version

0.18.0-77a66e7 (official ghcr.io/blakeblackshear/frigate:0.18.0-tensorrt, unmodified)

In which browser(s) are you experiencing the issue with?

N/A (backend)

Frigate config file

# Minimal excerpt; the full config has 40 cameras. A sanitized full copy can be shared privately.
detectors:
  onnx_0: {type: onnx}
  onnx_1: {type: onnx}
  onnx_2: {type: onnx}
  onnx_3: {type: onnx}
cameras:
  camera_a:  # ~18 cameras like this at 4K
    ffmpeg:
      hwaccel_args: preset-nvidia
      inputs:
        - path: rtsp://127.0.0.1:8554/camera_a
          roles: [detect, record]
    detect: {width: 3840, height: 2160, fps: 5}
  camera_b:  # the rest at 1080p or D1
    detect: {width: 704, height: 480, fps: 5}

docker-compose file or Docker CLI command

Kubernetes (k3s v1.35) Deployment, official image ghcr.io/blakeblackshear/frigate:0.18.0-tensorrt, NVIDIA runtime, 1 replica, tmpfs cache.

Relevant Frigate log output

No errors are logged. The drops are visible only as process_fps < camera_fps in /api/stats, while skipped_fps stays at 0.

Relevant go2rtc log output

N/A (go2rtc is healthy; the restreams deliver full frame rate)

Operating system

Other Linux (Ubuntu 24.04 VM, kernel 6.8)

Install method

Kubernetes (k3s), official image (not in the list)

Network connection

Wired

Camera make and model

Amcrest IP8M-2496E-V2 / IP8M-2796E-AI / IP8M-DLB2998EW-AI, Reolink Duo 3 / OMVI 3i (not camera-specific)

Screenshots of the Frigate UI's System metrics pages

Text instead of screenshots: CPU at most 60% of 48 vCPU (AMD Threadripper 2990WX VM, 4 NUMA nodes). GPU (RTX) detector inference 17–22 ms. The detected_frames_processor thread at 97% of one core.

Any other information that may be helpful

The same thread's synchronous full-resolution snapshot encode also makes object-end bursts on high-resolution cameras drop frames on every camera.

[NickM-27]

Please see https://github.com/blakeblackshear/frigate/blob/dev/AI_POLICY.md, these AI generated reports are not nearly as helpful as reports of actual issues with the direct information. The "suggestions" also lack knowledge of how Frigate works, used to work, and the reasons why it is designed the way it is.

Running high resolution detect is discouraged as it is in the vast majority of cases not beneficial and more likely to cause problems, in many ways:

• motion detection frame height is tuned for a max of 1920x1080
• the models are 320x320, so at a certain resolution the regions are downsized anyway, the resolution is not beneficial.

I ran the numbers on a few of my test setups and I see significantly less impact where the operation @ 3840x2160 takes between 0.3 and 0.6 ms instead of between 4.2 and 7.9 ms, looks like host or VM tuning would be the first recource.

In general, this is accepted by the design, losing a frame is not fatal or a major problem, if the system is overloaded this is acceptable behavior. The main problem here is that there is no way for the user to be aware of this. I will work on adding a system notice in 0.19 so the user will be made aware of this issue.

[NickM-27]

this is improved in #24619

[jvallery]

On high resolution detect, the goal is to try and capture better face coverage for facial recognition. I'm running the 640x640 frigate+ model. Any suggestions on how to maximize face recognition while running on the sub-stream?

I also have several 180 degree cameras which are mounted high up and have a very large coverage area. As a function of pixels, this makes potential objects very small without detect on higher resolutions.

[NickM-27]

1280x720 on a properly configured camera is good for short to mid distances, while 1920x1080 detection resolution is good for further distances. In most cases resolution for face recognition accuracy is not the issue, it is cameras that are incapable or improperly configured and have high motion blur.

Regarding 180 degree cameras, yes the resolution will be overall higher but still relative to that of the 16:9 cameras. I run mine at 2560x720 with no issues for LPR or face recognition.

[jvallery]

What do you mean by "improperly configured". Is there something specific to blur that you recommend? Are there camera level settings you recommend for maximizing face detection?

In general, my cameras have very basic settings. H.264, CBR, 5 FPS, native resolution for primary stream, max supported resolution for sub-stream, and minor image tuning based on location to compensate for night time headlight glare.

[NickM-27]

Yes, there are many camera settings which impact clarity for face recognition. Both stream-level (bit rate, constant bit rate streaming, etc.)

as well as expose settings, having proper shutter speeds and reduced noise reduction both help clarity as well.

[jvallery]

Related finding: the detected-frames queue holds only about 0.2 s, so any TrackedObjectProcessor stall drops frames on every camera.

• Queue size: app.py init_queues() creates detected_frames_queue = mp.Queue(maxsize=<enabled cameras> + 2). With 40 cameras at 5 fps that is 42 entries, about 0.2 s of the fleet's frames.
• Drop path: video/detect.py process_frames() drops the frame when the queue is full:

  if detected_objects_queue.full():
      frame_manager.close(frame_name)
      continue

  So process_fps falls below camera_fps with nothing logged, and skipped_fps stays at 0 because that counter tracks the capture-side frame_queue.
• Effect: any pause in the object-processor loop longer than about 0.2 s drops frames on every camera. We measured the synchronous full-resolution clean WebP snapshot encode in write_snapshot_to_disk() at 0.76–1.38 s per object end at 4K. Its time is nearly flat across snapshots.quality 50–90; JPEG would be about 40 ms. Each 4K object end therefore costs about 1 s of frames fleet-wide. In our data, 64.7% of object ends on 8 MP+ cameras were followed within 4 s by a fleet-wide drop burst, against 4.4% by chance.
• Memory is not the limit: the queue carries only frame names and detections. The shared-memory ring behind it already holds SHM_MAX_FRAMES (50) frames per camera, about 10 s at 5 fps here.

Suggestions (complementary to the ones above):

• Size the queue from aggregate fps, for example sum(detect.fps) × 2 s, capped below the SHM ring depth, or make it configurable.
• Move the snapshot encode to a worker, as suggested earlier.

Either would absorb these stalls; the loop itself averages about 55% busy here.