Native ONVIF cell-motion ingest

[widgetii]

1. Gap this addresses

• Frigate today has zero ONVIF event consumption. The onvif: camera block is bound exclusively to PTZ services (media / ptz / imaging in frigate/ptz/onvif.py).
• _init_onvif short-circuits at the get_definition("ptz") check (around line 198), so any camera without PTZ effectively cannot use the onvif: block at all — including all the analytics services the rest of the ONVIF Profile-M spec defines.
• Per-frame ImprovedMotionDetector runs on the decoded RTSP stream even when the camera already has hardware motion detection it would happily expose via ONVIF.

2. Why this matters

Many modern IP cameras have internal hardware IP blocks, which can efficiently detect motion between frames and help pipelining heavy GPU-based analytics (no motion in frame, no GPU inference is needed).

3. Standards stance

• Only ONVIF Profile-M standard topics and transports. No vendor-namespaced extensions on standard topics (e.g. no custom SimpleItems on VideoSource/MotionAlarm).
• Two transports consumed per ONVIF design intent:
  • WS-BaseNotification PullPoint for state (tns1:RuleEngine/CellMotionDetector/Motion IsMotion=true|false; also accept the legacy tns1:VideoSource/MotionAlarm State=true|false since many devices publish both).
  • RTSP analytics metadata stream (application/vnd.onvif.metadata) for per-frame cell coordinates (tt:MotionInCells, ONVIF Analytics Spec Annex B — base64 + PackBits, bit-packed row-major bitmap).
• Cell layout discovered via AnalyticsService.GetAnalyticsModules using the token returned by Media.GetVideoAnalyticsConfigurations — token format is implementation-defined per the WSDL, so it's looked up not assumed.

4. Design choices that deserve maintainer input

Each is presented as "chose X over Y because Z — open to redirection".

4.1 Module placement

• Chose: extend frigate/ptz/onvif.py — reuses the existing dedicated asyncio loop thread, ONVIFCamera client lifecycle, dynamic config-reload subscriber.
• Considered: new frigate/onvif/ package separating non-PTZ ONVIF surface.
• Cost of chosen: the PTZ module grows beyond PTZ. Cost of alternative: duplicate asyncio loop / reload infrastructure.

4.2 Analytics metadata stream consumption

• Chose: ffmpeg subprocess per enabled camera, -rtsp_transport tcp -map 0:d:0 -c copy -f data pipe:1, parse XML chunks from stdout.
• Considered: in-process async RTSP client (e.g. aiortsp).
• Reasoning: ffmpeg is already in deps and already manages per-camera streams; no async RTSP client in the existing dependency set handles application/vnd.onvif.metadata cleanly.
• Cost: extra low-bandwidth subprocess per enabled camera (~1 packet/sec at idle).

4.3 IPC main → camera subprocess

• Chose: manager.Value("b", 0) + manager.list() fields on existing CameraMetrics. Subscriber (main proc) writes; CameraTracker.process_frames reads.
• Considered: ZMQ pub/sub on a new topic, or a dedicated queue.
• Reasoning: CameraMetrics already follows this SyncManager-backed shared-state pattern for fps/pid counters; no new IPC surface.
• Cost: SyncManager marshalling overhead per read.

4.4 Event delivery model

• Chose: PullPoint only.
• Considered: also support WS-BaseNotification push (NotificationConsumer).
• Reasoning: PullPoint is NAT-friendly, no inbound port needed; simpler lifecycle (subscribe/pull/renew/unsubscribe).
• Cost: cameras that only publish push events not supported in v1 (rare in Profile-M).

4.5 Motion-source switch

• Chose: explicit motion.source: internal | onvif enum on MotionConfig.
• Considered: auto-derive from onvif.events.enabled.
• Reasoning: separates "I want camera-side events for monitoring" from "I want camera-side events to replace internal motion detection" — these are distinct user intents.
• Cost: two flags can desync; mitigated by validator requiring onvif.events.enabled=true when motion.source=onvif.

4.6 WSSE authentication scheme

• Surfacing as an open issue, not a chosen design.
• Today: OnvifConfig.tls_insecure: false (default) maps to python-onvif-zeep-async's encrypt=True, which sends WSSE PasswordDigest. A meaningful number of Profile-M devices reject Digest and only accept PasswordText (or HTTP Basic), and the only existing workaround is setting tls_insecure: true — conflating TLS verification with the WSSE auth scheme.
• This bites the events path specifically because every PullMessages call is authenticated.
• Open question: separate onvif.auth_mode: digest | text field, or accept the tls_insecure overload for v1?

4.7 Cell-bitmap decoder

• Chose: pure numpy + a small Python PackBits decoder + flood-fill connected-components.
• Considered: cv2.connectedComponentsWithStats.
• Reasoning: grid is small (a few hundred cells); cv2-free keeps the module unit-testable in isolation without OpenCV in the test env.
• Cost: ~30 lines of Python instead of one cv2 call.

4.8 Topic filter on subscription

• Chose: no filter — accept all notifications, parse for IsMotion / State SimpleItems.
• Considered: server-side Filter.TopicExpression narrowing the subscription to tns1:RuleEngine/CellMotionDetector/Motion.
• Reasoning: Profile-M devices vary in which standard topic they implement; same device often emits both. No filter = no chance of missing motion because the device chose the legacy topic.
• Cost: more notifications per Pull. Negligible at 1 Hz.

4.9 Configuration discovery flow

• Media.GetVideoAnalyticsConfigurations() → first config's token →
• Analytics.GetAnalyticsModules({"ConfigurationToken": <token>}) → find tt:CellMotionEngine module → extract CellLayout (Columns, Rows, Transformation).
• Falls back to no-events if any step fails.

4.10 Graceful degradation

• Camera advertises Analytics but no vnd.onvif.metadata track in the SDP → PullPoint binary state with full-frame motion box.
• Camera advertises events but update_xaddrs finds no analytics service → no events init, internal motion path unchanged.
• Camera's detector rule fires no transitions → motion stays off (correct; matches a camera with MD physically disabled).
• PackBits decode fails for a frame → that frame contributes nothing; PullPoint state still drives full-frame fallback.

4.11 Lifecycle & backward compatibility

• OnvifController.__init__ takes camera_metrics: dict[str, CameraMetrics] | None = None. Default None preserves all existing call sites.
• Per-camera teardown: stop_event.set() + task.cancel() + asyncio.wait_for(task, 5.0). Bounded so dynamic config reload stays responsive.
• Hot-reload on CameraConfigUpdateEnum.onvif already exists; the new tasks are torn down + respawned via the existing _close_camera → _init_single_camera chain.

5. Known limitations

• Metadata track typically bound to the primary RTSP profile only; sub-stream-only configs get binary-only motion with full-frame box.
• Single VideoAnalyticsConfiguration per camera assumed (first discovered is used).
• Some devices advertise the CellMotionDetector rule but their detector implementation never fires IsMotion=true — the integration correctly reflects this as "no motion" rather than working around camera-side bugs.

6. Reference implementation

• Reference impl at PR Native ONVIF cell-motion ingest #23354. Includes unit tests (PackBits round-trip on the spec example, cell-grid → box mapping, validator behaviour) and validator coverage for the new config combinations.
• Tested against OpenIPC and Xiongmai cameras (more vendor tests are welcome)

7. Asks for maintainer input

• Module placement: 4.1
• ffmpeg-subprocess vs new RTSP client dependency: 4.2
• motion.source enum on MotionConfig vs deriving from onvif.events.enabled: 4.5
• WSSE auth scheme — separate field or accept tls_insecure overload for v1: 4.6
• Scope of v1: PullPoint-only, or include WS-BaseNotification push: 4.4
• Anything in §4 you'd structure differently before reviewing the diff.

P.S. I apologize for the delay, but it’s better late than never.

[hawkeye217]

Thanks for the discussion.

Motion boxes in Frigate aren't a simple on/off gate for inference. They seed the detection regions that get cropped and scaled to the detector, so box tightness directly sets the effective resolution for small/distant objects. A 22×18 cell grid is roughly an order of magnitude coarser than the per-pixel contours our internal detector produces on the (already cheap) downscaled motion frame, and it's not tunable. The common fallback paths like binary-only cameras, no metadata track on the substream, and discovery/decode failure, all collapse to a full-frame box, which is the worst case for region seeding.

The bigger issue though is alignment. The ONVIF state and cells arrive out-of-band from a separate stream at a different time than the detect frame they'd be used to crop. Internal motion is computed from the exact frame at frame_time and this isn't. For anything moving, stale boxes place the region where the object was.

And I'm not sure the CPU argument holds, detection and inference are the expensive parts of Frigate's pipeline. Internal motion (frame-diff on a ~100px frame) is the cheap part you'd be replacing, while adding a third RTSP/ffmpeg subprocess per camera plus per-frame IPC list marshalling.

Curious if the other maintainers have thoughts.

[NickM-27]

Some specific concerns on top of those mentioned:

On top of what was said above, the coarser motion resolution also means that features which require this fine-grained motion, such as the region grid, may end up targeting the wrong area of the grid or using a larger region than we otherwise would have to detect a small object.

Latency

Adding this adds considerable latency, for both the transmission and interpretation of the message / polling frequency, as well as the time for the camera itself to process this motion. This can add up over time and also cause inconsistency for cameras as their load may go up.

Frigate strictly stays at real-time processing and a delay in motion data could cause the need to skip frames entirely once things start backing up.

No Ability to Tune

The existing motion tuner would be unusable, this makes for an inconsistent user experience as users have no way in Frigate to control how sensitive the motion is. This both affects general accuracy and capability to fine-tune, as well as the ability to use existing Frigate MQTT topics to dynamically tune based on the time of day or other factors.

UI Complexity

This would require a lot of reworking of the UI configuration to hide many settings depending on which motion mode is selected, which will require considerably more documentation and likely be confusing.

[NickM-27]

One other note, ONVIF experience shows us that cameras don't comply with the ONVIF spec quite often. This means that from camera to camera we could easily see different behavior and various bugs which we strictly make a policy of not wanting to implement per-camera quirks / fixes as we can't reliably maintain this.

It seems quite likely the motion data will have many differences between camera manufacturers and implementations which would make this considerably more complicated to maintain.

[widgetii]

Thanks both — the read is fair and a couple of the points landed harder than the framing I led with.

Where you're right:

• Boxes as region seeds, not gates. I framed motion as state-with-coordinates and underweighted that the box drives the cropped/scaled region the detector actually runs on. A coarse box for that role is genuinely worse than no change — that's not something we can spin around.
• Temporal alignment. The cell stream is out-of-band by design and there's no clean fix that survives vendor variance. Conceded.
• Vendor compliance and maintenance burden. Legit. Not something the design on our side can fix.
• UI / tuner / region-grid coupling. All real if motion.source becomes a mode switch.

Two points I'd push back on, mostly for clarity:

• The CPU argument is workload-dependent. It's accurate for a busy scene on a host where decode + frame-diff is already negligible. The case I had in mind is the opposite: a fixed camera on a mostly-empty space (corridor, equipment room, parking spot) with reliable on-camera MD. In that pattern, current Frigate decodes every frame and runs motion on them, all day, for nothing. The saving isn't visible on a beefy x86 but it's real on edge hosts.
• The 22×18 coarseness comes from ONVIF, not from us. Cameras do have finer data internally, but the spec only transports what CellLayout describes — there's no per-pixel coordinate channel at the Analytics layer to consume. So agreeing the box is too coarse for region seeding is agreeing with the conclusion, but the cause is the wire format, not a design choice we made.

Narrowing question — would a much smaller scope be more interesting?

• PullPoint binary state as a detector-gate only, no motion-box replacement.
• Internal motion detection stays exactly as-is: same tuner, same region grid, same UI, same MQTT surface.
• When the camera reports IsMotion=false for the configured debounce window, Frigate skips the detection pipeline (region selection + inference) entirely. When the camera reports motion, internal motion runs normally and seeds boxes the way it does today.
• No cell grid, no vnd.onvif.metadata RTSP track, no IPC list marshalling. Just the PullPoint state on OnvifController's existing loop.
• This collapses roughly to ~1/4 the diff and sidesteps the alignment, region-grid, tuner, and UI-mode-switching concerns — none of those surfaces change.

If even that scope isn't a fit for Frigate's architecture, totally fine — happy to maintain the larger thing out-of-tree. Wanted to ask before withdrawing.

[NickM-27]

but the cause is the wire format, not a design choice we made.

Sure, but that is sort of unimportant distinction, the simple fact is it will be a degraded level of information.

PullPoint binary state as a detector-gate only, no motion-box replacement.

This seems to be making the assumption that the motion detection load is static, but it is not. when a scene is static and does not have activity occurring, motion detection is very lightweight as cv2 has much less work to do, so the CPU load is minimal.

if we gate on the camera we still incur the latency and maintenance complexity, only to save the occasional bit of front-load work not running motion detection.

This approach also won't really work as is. The motion detector saves an average of the frames to keep a constant-state motion going.

So if we gate on a period of time where no significant motion happened, but there were gradual changes. Then when motion is detected by the camera we will have significant motion detection occur in Frigate's motion detector as it will not have saved averages of the slight changes in the frame over that time. Leading to much larger and less accurate motion boxes.

(a practical example of this is clouds moving, sun changing, etc.)

[NickM-27]

I think to justify this as an actual problem, we would need to start by seeing some static re-testable videos and seeing what the motion detection load truly is. For example I run a handful of streams on my Orange Pi for testing of the RKNN detector and motion detection is not a considerable load compared to video decoding or object tracking. That is about as far on the edge as it really makes sense to go.