Bug: norfair distance() "Received nan values from distance function" kills the camera process during PTZ autotracking (healthy detector); process never restarts

[pgregg88]

Summary

On an autotracking PTZ camera with a healthy Coral EdgeTPU detector, the per-camera tracking process dies intermittently (~2-3×/month) with:

ValueError: Received nan values from distance function, please check your distance function for errors!

This is the same error string as #9742, but a different root cause. In #9742 the NaN came from a broken TensorRT model on Maxwell/compute-5.0 Quadro cards returning bad coordinates, and was treated as a GPU/model problem (users moved to newer GPUs; closed as stale). Here the detector is healthy and detections are normal — the degenerate box originates in the tracker/autotracking path (a collapsed Kalman estimate or a box clamped to zero width/height during camera motion). The common, generalizable defect is that distance() has no guard against a zero dimension, so any degenerate box — whatever its origin — crashes the whole camera process.

Root cause

frigate/track/norfair_tracker.py::distance() (unchanged on dev as of this writing):

estimate_dim = np.diff(estimate, axis=0).flatten()
detection_dim = np.diff(detection, axis=0).flatten()
...
distance[0] /= estimate_dim[0]          # ZeroDivision -> nan when estimate width  == 0
distance[1] /= estimate_dim[1]          # ZeroDivision -> nan when estimate height == 0
widths  = np.sort([estimate_dim[0], detection_dim[0]])
heights = np.sort([estimate_dim[1], detection_dim[1]])
width_ratio  = widths[1]  / widths[0]  - 1.0   # ZeroDivision -> nan when smaller width  == 0
height_ratio = heights[1] / heights[0] - 1.0   # ZeroDivision -> nan when smaller height == 0

When a dimension is 0, the division yields inf/nan; norfair's Tracker._update_objects_in_place then raises ValueError, and because this runs inside process_frames, the camera process exits.

Two problems

1. Prevent: guard the divisions. Clamp box dims to ≥1px (pixel coordinates, so <1 is meaningless):

   estimate_dim  = np.maximum(np.abs(np.diff(estimate,  axis=0).flatten()), 1.0)
   detection_dim = np.maximum(np.abs(np.diff(detection, axis=0).flatten()), 1.0)

   This single change protects all four division sites and removes the crash class regardless of why the box is degenerate (covers [Detector Support]: v13, "Received nan values from distance function" #9742's symptom too). Happy to open a PR.

2. Recover: independent of the math, an unhandled exception in process_frames should not leave a camera permanently dark. The killed frigate.process:<cam> is not respawned (we observed ~15h); meanwhile frigate.capture:<cam> keeps running, so camera_fps stays > 0 and /api/stats keeps reporting the dead process's pid (misleading). Consider catching/skipping a single bad tracker frame and/or having the watchdog restart a dead detect process. Relates to the long-standing Frigate continue working after detector not available #2904.

Version

0.17.1-416a9b7 (stock ghcr.io/blakeblackshear/frigate:0.17.1, unmodified). norfair 2.3.0, Python 3.11.2.

Config (redacted)

detectors:
  coral:
    type: edgetpu
model:
  model_type: yolo-generic
  width: 320
  height: 320
cameras:
  lake_cam_ptz:
    detect: { width: 1280, height: 720, fps: 5 }
    objects: { track: [bird, boat, person] }
    onvif:
      autotracking:
        enabled: true
        track: [bird, boat, person]
        return_preset: BoatsNorthEastZoom

Log output (the crash)

/opt/frigate/frigate/track/norfair_tracker.py:58: RuntimeWarning: divide by zero encountered in scalar divide
  distance[1] /= estimate_dim[1]
/opt/frigate/frigate/track/norfair_tracker.py:64: RuntimeWarning: divide by zero encountered in scalar divide
  width_ratio = widths[1] / widths[0] - 1.0
/opt/frigate/frigate/track/norfair_tracker.py:56: RuntimeWarning: invalid value encountered in scalar divide
  distance[0] /= estimate_dim[0]
Process frigate.process:lake_cam_ptz:
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/frigate/frigate/video.py", line 662, in run
    process_frames(
  File "/opt/frigate/frigate/video.py", line 983, in process_frames
    object_tracker.match_and_update(
  File "/opt/frigate/frigate/track/norfair_tracker.py", line 585, in match_and_update
    tracked_objects = self.default_tracker[mode].update(
  File "/usr/local/lib/python3.11/dist-packages/norfair/tracker.py", line 212, in update
    ) = self._update_objects_in_place(
  File "/usr/local/lib/python3.11/dist-packages/norfair/tracker.py", line 297, in _update_objects_in_place
    raise ValueError(
ValueError:
Received nan values from distance function, please check your distance function for errors!

After this, there is no camera-process restart in the logs; frigate.process:lake_cam_ptz is absent from the container process list while frigate.capture:lake_cam_ptz keeps running.

Notes

Reproduction is intermittent and tied to autotracking motion, so it's hard to force on demand, but the divide-by-zero is deterministic given a zero-dim estimate/detection. We run a live autotracking PTZ and are happy to test a patched norfair_tracker.py to confirm the guard eliminates the crash over time.

[inkeep-ai-assistant[bot]]

This is a well-documented bug report identifying a divide-by-zero vulnerability in the Norfair tracker's distance function during PTZ autotracking. The issue occurs when box dimensions collapse to zero (from Kalman estimate drift or clamping during camera motion), causing NaN values that crash the camera process.

Your proposed fix is sound—clamping dimensions to ≥1px would protect all four division sites:

estimate_dim  = np.maximum(np.abs(np.diff(estimate,  axis=0).flatten()), 1.0)
detection_dim = np.maximum(np.abs(np.diff(detection, axis=0).flatten()), 1.0)

Regarding the autotracking configuration, the documentation confirms that calibration can help with tracking accuracy during PTZ movements. (1) If you haven't already, you might try running calibration by setting calibrate_on_startup: True and restarting Frigate—this measures PTZ motor speeds and helps estimate object positions after camera moves. (1) After calibration completes, the config is automatically updated with movement_weights, and you should set calibrate_on_startup back to False. (1)

The documentation also notes that the motion estimator needs sufficient scene details to track camera motion—if zoom_factor is too high or the scene lacks hard edges/color variations, tracking can fail. (1) However, this wouldn't explain the zero-dimension box crash you're experiencing.

For the process recovery issue (problem #2), this appears to be outside what's documented in the available sources. Opening a PR with the guard fix and filing the process restart issue separately would be the recommended path forward.

📚 Sources:

1. Camera Autotracking

---

Have another question? Just tag @inkeep.

[hawkeye217]

This rare crash is likely due to autotracking motion estimation in difficult circumstances. The PR your AI put up only clamps the divisors, but the crash likely comes from a non-finite estimate, not a zero-area box. I've pushed some changes to improve this in #23475

Let me know if you still have issues after this fix.

[pgregg88]

Thanks. I have the proposed fix deployed and it held today. I'll keep a look out for what you suggested and report back what I find.

I'm definitely working in difficult circumstances. I'm tracking boats and watercraft at distance on a busy lake.

I appreciate you looking into this. -Preston

[hawkeye217]

Great, glad to hear. I'll close this for now, but feel free to comment or reopen if the issue recurs.