Skip to content

AI Detection

Jump to bottom
wiki-sync-bot edited this page Jul 26, 2026 · 1 revision

AI Detection

Overview

YOLOv8n ONNX object detection on camera sub-streams with MOG2 motion gating, zone filtering, and position-aware deduplication.

Pipeline

RTSP Sub-stream (0.5 FPS, 640px width)
            │
            ▼
┌───────────────────┐
│  FrameSampler       │  OpenCV cap.read()
│  per camera         │  cv2.CAP_PROP_BUFFERSIZE=1 (minimal latency)
└────────┬──────────┘
            │
            ▼
┌───────────────────┐
│  MOG2 Motion Gate │  history=500, detectShadows=False
│   (OpenCV)           │  countNonZero > 500 pixels → motion
│                     │  Threshold: low=40, med=25, high=16
└────────┬──────────┘
            │ motion detected
            ▼
┌───────────────────┐
│  YOLOv8n ONNX       │  Shared singleton session
│   (ONNX Runtime)     │  Intra-op threads: 1
│  INPUT_SIZE=640     │  CPUExecutionProvider
└────────┬──────────┘
            │
            ▼
┌───────────────────┐
│  Post-processing    │  Transpose (1,84,8400) → per-class argmax
│                     │  NMS (IoU=0.45), confidence clamp [0.05, 0.95]
│  Letterbox unscale│
└────────┬──────────┘
            │
            ▼
┌───────────────────┐
│  Zone Filter        │  cv2.pointPolygonTest on bbox bottom-center
│  Confidence check │  Only objects inside zones pass
└────────┬──────────┘
            │
            ▼
┌───────────────────┐
│  Position Dedup     │  Static object: 1 event/5min (STATIC_COOLDOWN_S)
│                     │  Moved (>0.10 normalized): immediate event
│  Min event gap: 5s │  per class (MIN_EVENT_GAP_S)
└────────┬──────────┘
            │
       ┌────┴────┐
       ▼           ▼
   Events     JPEG
   Table      Snapshot

Per-Camera Configuration

Config Type Description
ai_enabled boolean Enable/disable AI detection per camera
ai_objects JSON array COCO classes to detect: ["person", "car", "truck", "bus", "motorcycle", "bicycle", ...] (80 total)
ai_zones JSON array Polygon zones: [{"name": "...", "points": [[x,y], ...]}] — empty zones = whole frame
ai_sensitivity string MOG2 threshold: low (40), medium (25), high (16)
ai_min_confidence float Confidence threshold, clamped 0.05–0.95

Performance Optimization (v0.01.15)

Change Before After Impact
AI FPS 2 FPS 0.5 FPS CPU: 166% → 74%
ONNX threads 2 1 Reduced contention
MOG2 sensitivity medium low Fewer frames reach YOLO
Sub-stream bitrate 2000k 1000k Lighter FFmpeg encode
AI engine CPU limit 4 cores 2 cores cgroup enforcement

Result: Load avg 41→21 (-48%), AI RAM 955MB→565MB (-41%)

Motion-Only Mode

Cameras with recording_mode='motion' and ai_enabled=False get a motion-only sampler:

  • MOG2 gate only (no YOLO inference)
  • Publishes motion state to Redis nvr:motion
  • Recording Engine's MotionRecorderController starts/stops FFmpeg based on motion state
  • Motion active → start recorder; motion inactive 30s → stop recorder

Worker Lifecycle

  • Reconcile interval: 15s (POLL_INTERVAL)
  • Config change detection: zones, objects, stream URI changes trigger worker recreation within 15s
  • Model: yolov8n.onnx in ai_models volume at /app/models (exported on host via ultralytics)

Event Output

  • Events table: Plain SQL insert with metadata JSON {objects: [{label, confidence, box}], zone_id, ...}
  • JPEG snapshots: Saved to /data/recordings/snapshots/ with token auth
  • Redis pub/sub: nvr:events channel for real-time broadcast

Clone this wiki locally