Skip to content

Troubleshooting

Jump to bottom
wiki-sync-bot edited this page Jul 26, 2026 · 1 revision

Troubleshooting

1. CSP Blocks hls.js in RecordingPlayer (Chrome)

Symptom: Recordings page shows spinner forever, no errors in console (Chrome). Works in Safari.

Root cause: hls.js uses new Function() internally. CSP script-src blocks unsafe-eval. Chrome enforces CSP strictly; Safari doesn't.

Fix: RecordingPlayer only plays MP4 (progressive download via /api/v1/recordings/{id}/stream), NOT HLS. No hls.js in RecordingPlayer. Do NOT add hls.js to RecordingPlayer.

2. Chrome vs Safari Video Playback Differences

Chrome is strict about:

  • Range: bytes=0- → MUST return 206 Partial Content (Safari accepts 200 OK)
  • pix_fmt=yuvj420p full-range color → rejects (Safari handles)
  • CSP unsafe-eval → blocks (Safari ignores)

3. Recording Codec Normalization

Problem: Cameras may output HEVC or H.264 with pix_fmt=yuvj420p color_range=pc. Both cause Chrome playback failure.

Fix in recording engine:

  • HEVC → transcode to H.264: -c:v libx264 -preset ultrafast -crf 20 -pix_fmt yuv420p
  • H.264 → apply bitstream filter: -bsf:v h264_metadata=video_full_range_flag=0
  • Codec detected via ffprobe before FFmpeg command is built

4. HTTP Range Request Handling

Problem: Chrome sends Range: bytes=0- for video pre-flight. Returning 200 OK causes Chrome to reject/drop the stream.

Fix in recordings endpoint:

  • Parse Range header via _parse_range()
  • Check is_full_range (start==0, end==file_size-1)
  • Full range → 200 OK (progressive download, browser buffers as it streams)
  • Partial range → 206 Partial Content + Content-Range (seeking)

5. Stream Endpoint Content-Disposition

Content-Disposition: inline on video responses interferes with browser playback. Only use attachment for ?download=true.

6. FFmpeg Startup Crashes (HLS Stability)

Problem: FFmpeg crashes every few seconds on stream relay startup.

Fix (v0.01.14):

  • -stimeout → -timeout (FFmpeg 5.1.9 doesn't support -stimeout)
  • Removed -reconnect flags (not supported for RTSP in FFmpeg 5.1.9)
  • Monitor loop: returncode=0 clean exits bypass circuit breaker, 5 quick reconnect attempts
  • Monitor loop: returncode≠0 errors get 3 attempts with transport fallback

7. Stream Relay Lifecycle

  1. Frontend calls POST /api/v1/cameras/{id}/live/start?stream=main|sub
  2. live_relay.py checks stream-manager status first, then delegates
  3. Stream-manager starts FFmpeg with libx264 transcode → pushes to MediaMTX
  4. MediaMTX creates HLS at rtsp://nvr-mediamtx:8554/{camera_id}
  5. Frontend polls /hls/{cid}/index.m3u8 then initializes HLS.js
  6. Circuit breaker prevents rapid restarts (15s→120s cooldown for stream relay)
  7. Idle reaper: stream-manager stops relays with zero MediaMTX readers for 10 min (STREAM_IDLE_TIMEOUT_S)

8. High CPU / Load Issues

Symptom: Server load avg > 30, AI RAM > 900MB

Fix (v0.01.15):

  • Reduce AI FPS from 2 to 0.5
  • Reduce ONNX threads from 2 to 1
  • Lower MOG2 sensitivity from medium to low
  • Reduce sub-stream bitrate from 2000k to 1000k
  • Limit AI engine Docker CPU to 2 cores

Result: Load avg 41→21 (-48%), AI RAM 955MB→565MB (-41%)

Quick Reference: Key Files

File Purpose
services/stream-manager/app/manager.py FFmpeg process lifecycle, circuit breaker, idle reaper
services/stream-manager/app/relay_api.py HTTP API for relay start/stop/status
services/api/app/services/live_relay.py Stream relay delegation to stream-manager
services/recording-engine/app/main.py Recording loops (reconcile/catalog/retention/analytics)
services/recording-engine/app/recorder.py Per-camera FFmpeg supervisor with auto-restart
services/recording-engine/app/catalog.py Segment→DB registration + sync
services/recording-engine/app/retention.py Circular + age-based cleanup
services/ai-engine/app/main.py AI worker reconcile loop
services/ai-engine/app/detector.py YOLOv8 ONNX post-processing + NMS
services/ai-engine/app/frame_sampler.py Motion gate + dedup + event persist

Clone this wiki locally