feat(api): surface motion-detector-down state in decode-status + admin console - #415
Merged
Conversation
…n console On a multi-GPU host a driver reorder pointed the recorder's VAAPI motion decode at the wrong render node, so the pixel detector produced zero frames. The recorder correctly failed OPEN (recorded continuously so no footage was lost), but the fault ran days undetected: fail-open continuous segments look identical to a busy scene, and an operator viewing the admin console had no way to SEE that a camera's motion detection was down and recording continuously as a fallback (issue #411 item 2). The push-alert path already exists (recorder emits a motion_detector_unhealthy system_event; migration 0038 seeds a default-enabled rule; notifications.rs delivers it over configured channels). This adds the missing IN-APP signal with no push channel required. Represent CURRENT health as a LEVEL, not an EDGE. The system_event is an edge with no "healthy again" marker, so events alone cannot answer "is this camera unhealthy right now". New table camera_motion_health (migration 0072) holds exactly one row per camera, upserted by the recorder's health aggregator (motion.rs aggregate_health) on every camera-level transition in either direction, so it always reflects the current state. changed_at advances only on a flip (reads as "recording continuously since <time>"); the row is deleted when a camera's worker stops (mirrors camera_decode_status) and cascades on camera delete. The in-memory fail-open signal still flips first and unconditionally; the DB write is best-effort telemetry and never gated to it. GET /config/decode-status LEFT JOINs the row and returns motion_healthy / motion_health_since / motion_health_reason; admin.html renders a var(--danger) "Motion detection DOWN, recording continuously as fallback (for <elapsed>)" badge in the decode-status panel, nothing when healthy or unreported. Notification delivery path verified clean: motion_detector_unhealthy is recognized at the rule lookup, enabled, cooldown 900s honored, has a human label. No default channel shipped (that would be phone-home). Deferred follow-ups: desktop, Android, and iOS do not yet surface the new per-camera field. Tests: unit tests for the fail_open_reason derivation (single/multiple/mixed/ all-healthy). The existing aggregate_health test gains a lazy unreachable pool so the best-effort upsert fails fast and is swallowed. Signed-off-by: badbread <badbread@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up #411 (item 2: distinguish "fail-open because the detector is broken" from "recording because there is motion", and make it visible in-app). Emission of the alert (#413) and the default-enabled notification rule (migration 0038) already exist; this adds the missing IN-APP signal, no push channel required.
The gap
When a motion detector goes down, the recorder correctly fails open (records continuously so no footage is lost). But fail-open continuous segments look identical to a busy scene, and an operator viewing the admin console had no way to SEE that a camera's motion detection was down and recording continuously as a fallback. That is exactly why the #411 incident ran days undetected.
Why a new LEVEL signal
The existing
motion_detector_unhealthysystem_event is an EDGE with no "healthy again" marker, so events alone cannot answer "is this camera unhealthy right now". New tablecamera_motion_health(migration0072, registered) holds exactly one row per camera, upserted by the recorder's health aggregator (aggregate_health) on every camera-level transition in either direction, so it always reflects current state.changed_atadvances only on a flip (reads as "recording continuously since "); the row is deleted when a camera's worker stops (mirrorscamera_decode_status) and cascades on camera delete.Recorder-correctness note
The in-memory fail-open signal (
health_tx.send) still flips FIRST and unconditionally; thecamera_motion_healthwrite is best-effort telemetry after it, errors swallowed at DEBUG, and never gates recording. Teardown-unhealthy is deliberately NOT persisted (worker shutdown is not a detector fault; the supervisor deletes the row instead). No change to what gets recorded or to the fail-open path.Surfacing
GET /config/decode-statusLEFT JOINs the row and returnsmotion_healthy/motion_health_since/motion_health_reason.admin.htmlrenders avar(--danger)"Motion detection DOWN, recording continuously as fallback (for )" badge in the decode-status panel; nothing when healthy or unreported.Notification path
Verified clean end-to-end:
motion_detector_unhealthyis recognized at the rule lookup, the0038rule is enabled, cooldown 900s is honored, and it has a human label. No default channel shipped (that would be phone-home).Deferred follow-ups
Desktop (Flutter), Android, and iOS do not yet surface the new per-camera field. Noted in
COMPONENT-MAP.md. Pure-Frigate/HA-only motion cameras with no local decode row do not appear in the decode-status panel, so their badge would not show there; the common pixel path and the #411 case are covered.Testing
Unit tests for
fail_open_reason(single / multiple / mixed / all-healthy). The existing aggregate_health test gains a lazy unreachable pool so the best-effort upsert fails fast and is swallowed. Full gate runs in CI.node --checkon the extracted admin.html script passed.