Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Support]: Transcoded stream trips up ffmpeg #2334

Closed
brujoand opened this issue Nov 27, 2021 · 15 comments
Closed

[Support]: Transcoded stream trips up ffmpeg #2334

brujoand opened this issue Nov 27, 2021 · 15 comments

Comments

@brujoand
Copy link

Describe the problem you are having

So I'm trying to force my horde of Reolink rlc-820a cameras to play ball with Frigate. I've set up a pod which is transcoding the streams with ffmpeg and rtsp-simple-server like so:

    paths:
      stua:
        runOnInit:
          ffmpeg -hide_banner -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt+igndts -rtsp_transport tcp -stimeout 5000000 -analyzeduration 2000 -probesize 2000 -hwaccel qsv -c:v hevc_qsv -r:v 20/1 -vsync vfr -i 'rtsp://admin:password@10.20.1.14:554//h265Preview_01_main' -c:v h264_qsv -c:a copy -f rtsp -rtsp_transport tcp rtsp://0.0.0.0:8554/stua -vf 'format=qsv,fps=fps=5,scale_qsv=1280:720' -c:v h264_qsv -c:a copy -f rtsp -rtsp_transport tcp rtsp://0.0.0.0:8554/stua_sub
        runOnInitRestart: yes
      stua_sub:
        source: publisher

So basically I've got the camera streaming h265(4k, 20fps) video to a transcoder pod, which decodes h265 and creates a main stream of h264(4k, 20fps) and a substream of h264(720p,5pfs). I can view these two transcoded streams just fine with VLC, but frigate refuses to even aknowledge that video is being recieved:

No frames received from stua in 20 seconds. Exiting ffmpeg...

So I'm assuming that there is something wrong about the way I'm re-encoding the stream that I'm feeding into Frigate, but I can't quite figure out what. Any advice?

Version

0.9.4-26ae608

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: pci
mqtt:
  host: vernemq-0.vernemq-headless.data.svc.cluster.local
  topic_prefix: frigate
database:
  path: /data/frigate.db
objects:
  track:
    - person
    - bil
    - skapning
  filters:
    person:
      threshold: 0.7
    skapning:
      threshold: 0.7
detect:
  max_disappeared: 50
model:
  labelmap:
    2: bil
    5: bil
    7: bil
    15: skapning
    16: skapning
    17: skapning
    18: skapning
    19: skapning
    20: skapning
    22: skapning
rtmp:
  enabled: False
cameras:
  stua:
    ffmpeg:
      inputs:
        - path: rtsp://transcoding.automation.svc.cluster.local:8554/stua_sub
          roles:
            - detect
        - path: rtsp://transcoding.automation.svc.cluster.local:8554/stua
          roles:
            - record
    snapshots:
      enabled: false
    record:
      enabled: false
    detect:
      height: 720
      width: 1280
      fps: 5

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2021-11-27 16:23:41] frigate.app                    INFO    : Starting Frigate (0.9.4-26ae608)
[2021-11-27 16:23:41] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2021-11-27 16:23:41] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2021-11-27 16:23:41] peewee_migrate                 INFO    : There is nothing to migrate
[2021-11-27 16:23:42] frigate.mqtt                   INFO    : MQTT connected
[2021-11-27 16:23:42] detector.coral                 INFO    : Starting detection process: 216
W driver/beagle/beagle_kernel_top_level_handler.cc:131] Could not set performance expectation : 36 (Inappropriate ioctl for device)
[2021-11-27 16:23:42] frigate.edgetpu                INFO    : Attempting to load TPU as pci
[2021-11-27 16:23:42] ws4py                          INFO    : Using epoll
[2021-11-27 16:23:42] frigate.app                    INFO    : Output process started: 218
[2021-11-27 16:23:42] frigate.edgetpu                INFO    : TPU found
[2021-11-27 16:23:42] frigate.app                    INFO    : Camera processor started for stua: 224
[2021-11-27 16:23:42] frigate.app                    INFO    : Capture process started for stua: 227
[2021-11-27 16:23:42] ws4py                          INFO    : Using epoll
[2021-11-27 16:24:02] watchdog.stua                  INFO    : No frames received from stua in 20 seconds. Exiting ffmpeg...
[2021-11-27 16:24:02] watchdog.stua                  INFO    : Waiting for ffmpeg to exit gracefully...
[2021-11-27 16:24:33] watchdog.stua                  INFO    : FFmpeg didnt exit. Force killing...
[2021-11-27 16:24:33] frigate.video                  INFO    : stua: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2021-11-27 16:24:33] frigate.video                  INFO    : stua: ffmpeg process is not running. exiting capture thread...
[2021-11-27 16:24:43] watchdog.stua                  ERROR   : FFMPEG process crashed unexpectedly for stua.
[2021-11-27 16:24:43] watchdog.stua                  ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-27 16:24:43] watchdog.stua                  ERROR   : You may have invalid args defined for this camera.
[2021-11-27 16:24:43] ffmpeg.stua.detect             ERROR   : [h264 @ 0x557a28beaf40] co located POCs unavailable

FFprobe output from your camera

### Source stream
[rtsp @ 0x560b325afb80] UDP timeout, retrying with TCP
Input #0, rtsp, from 'rtsp://admin:password@10.20.1.14:554//h265Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: hevc, yuv420p(tv), 3840x2160, 24.92 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

### Main stream
[h264 @ 0x7fec3901cc00] co located POCs unavailable
Input #0, rtsp, from 'rtsp://10.30.0.5:8554/stua':
  Metadata:
    title           : Stream
  Duration: N/A, start: -0.122625, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 3840x2160 [SAR 1:1 DAR 16:9], 20 fps, 20 tbr, 90k tbn, 40 tbc
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

### Sub stream
[h264 @ 0x7fa5be00a000] co located POCs unavailable
Input #0, rtsp, from 'rtsp://10.30.0.5:8554/stua_sub':
  Metadata:
    title           : Stream
  Duration: N/A, start: -1.051644, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 5 fps, 5 tbr, 90k tbn, 10 tbc
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Frigate stats

{
  "detection_fps": 0,
  "detectors": {
    "coral": {
      "detection_start": 0,
      "inference_speed": 10,
      "pid": 216
    }
  },
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 16750.5,
        "mount_type": "tmpfs",
        "total": 16750.8,
        "used": 0.3
      },
      "/media/frigate/clips": {
        "free": 4564372.2,
        "mount_type": "nfs4",
        "total": 11615690.3,
        "used": 7051318.1
      },
      "/media/frigate/recordings": {
        "free": 4564372.2,
        "mount_type": "nfs4",
        "total": 11615690.3,
        "used": 7051318.1
      },
      "/tmp/cache": {
        "free": 168012.8,
        "mount_type": "overlay",
        "total": 210304.5,
        "used": 31537.5
      }
    },
    "uptime": 11563,
    "version": "0.9.4-26ae608"
  },
  "stua": {
    "camera_fps": 0,
    "capture_pid": 226,
    "detection_fps": 0,
    "pid": 223,
    "process_fps": 0,
    "skipped_fps": 0
  }
}

Operating system

Other Linux

Install method

Docker CLI

Coral version

M.2

Network connection

Wired

Camera make and model

RLC-820a

Any other information that may be helpful

No response

@brujoand
Copy link
Author

brujoand commented Nov 27, 2021

The only thing that seems wrong to me is the discrepancy between the ffprobe output above and the ffmpeg output from the transcoder pod:

Input #0, rtsp, from 'rtsp://admin:password@10.20.1.14:554//h265Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv), 3840x2160, 90k tbr, 90k tbn, 90k tbc
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> h264 (h264_qsv))
  Stream #0:1 -> #0:1 (copy)
  Stream #0:0 -> #1:0 (hevc (hevc_qsv) -> h264 (h264_qsv))
  Stream #0:1 -> #1:1 (copy)
Press [q] to stop, [?] for help
Output #0, rtsp, to 'rtsp://0.0.0.0:8554/stua':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264, qsv(tv, progressive), 3840x2160, q=2-31, 1000 kb/s, 20 fps, 90k tbn
    Metadata:
      encoder         : Lavc58.134.100 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Output #0, rtsp, to 'rtsp://0.0.0.0:8554/veien':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264, qsv(tv, progressive), 3840x2160, q=2-31, 1000 kb/s, 20 fps, 90k tbn
    Metadata:
      encoder         : Lavc58.134.100 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Output #1, rtsp, to 'rtsp://0.0.0.0:8554/stua_sub':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
    encoder         : Lavf58.76.100
  Stream #1:0: Video: h264, qsv(tv, progressive), 1280x720, q=2-31, 1000 kb/s, 5 fps, 90k tbn
    Metadata:
      encoder         : Lavc58.134.100 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
  Stream #1:1: Audio: aac (LC), 16000 Hz, mono, fltp

So here it states that the transcoded streams are qsv(tv, progressive) but ffprobe claims it is yuv420p(progressive). Not sure if that's important at all, I'm fairly new to video and ffmpeg.

@brujoand
Copy link
Author

brujoand commented Nov 27, 2021

So I think there might be two problems here.

  1. The stream is slow to start, even on vlc and it often takes 20-30 seconds before I see an actual image
  2. Sometimes it seems that Frigate can start the stream before the 20 second timout and I get the following errors:
[2021-11-27 17:16:40] watchdog.stua                  ERROR   : FFMPEG process crashed unexpectedly for stua.
[2021-11-27 17:16:40] watchdog.stua                  ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-27 17:16:40] watchdog.stua                  ERROR   : You may have invalid args defined for this camera.
[2021-11-27 17:16:40] ffmpeg.stua.detect             ERROR   : [h264 @ 0x55b162e33f40] co located POCs unavailable
[2021-11-27 17:16:40] ffmpeg.stua.detect             ERROR   : [rtsp @ 0x55b162e2e780] DTS discontinuity in stream 1: packet 55 with DTS 26208533506437, packet 56 with DTS 26208533508504
[2021-11-27 17:16:40] ffmpeg.stua.detect             ERROR   : [rtsp @ 0x55b162e2e780] DTS discontinuity in stream 1: packet 57 with DTS 26208533508505, packet 58 with DTS 26208533511322

So I'm guessing it fails to start within 20 seconds most of the time, and when it does. Well the above happens which I'm not quite sure about.

@brujoand
Copy link
Author

brujoand commented Nov 27, 2021

Ha! Setting -adaptive_i 1 sped up the load time of the stream and now Frigate finally gets it! :D

A bit too early to celebrate it seems. The stream worked fine in Frigate and was snappy to load in vlc, so I re-enabled my 5 other cameras (3 reolink and 2 hikvision) and now I'm back to the same errors. I guess I just got lucky once :/

@brujoand
Copy link
Author

Okay, finally got this working by moving from qsv to vaapi:

ffmpeg -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -hide_banner -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt+igndts -rtsp_transport tcp -stimeout 5000000 -analyzeduration 2000 -probesize 2000 -hwaccel vaapi -r:v 20/1 -vsync vfr -i 'rtsp://admin:password@10.20.1.14:554//h265Preview_01_main' -vf 'fps=5,scale_vaapi=w=1280:h=720:format=nv12' -c:v h264_vaapi -movflags +faststart -c:a copy -f rtsp-rtsp_transport tcp rtsp://0.0.0.0:8554/stua_sub -c:v h264_vaapi -movflags +faststart -c:a copy -f rtsp -rtsp_transport tcp rtsp://0.0.0.0:8554/stua

This works perfectly and the transcoder is not even using half the capacity of my quicksync gpu when doing this for 3 cameras at the time.

So, I was using this approach with a transcoder pod to get a better feedback loop when developing the ffmpeg command. Now though I'm uncertain about how to best implement this. So the Reolink cameras have a crappy sub stream which isn't good enough for detection in my case, which is why I nead to create two outputs. Would I be able reuse this ffmpeg command in frigate to create both the sub stream and main stream in one go? In other words without fetching the camera stream twice?

@brujoand
Copy link
Author

brujoand commented Nov 28, 2021

So the above worked, but kept crashing every few minutes. And the stream was still fairly slow to load. This seems to have been the final part of the puzzle:

ffmpeg -threads 1 -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -hide_banner -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt+igndts -rtsp_transport tcp -stimeout 5000000 -analyzeduration 2000 -probesize 2000 -hwaccel vaapi -r:v 20/1 -vsync vfr -i 'rtsp://admin:password@10.20.1.14:554//h265Preview_01_main' -vf 'fps=5,scale_vaapi=w=1280:h=720:format=nv12' -c:v h264_vaapi -force_key_frames "expr:gte(t,n_forced*5)" -movflags +faststart -an -f rtsp -rtsp_transport tcp rtsp://0.0.0.0:8554/stua_sub -c:v h264_vaapi -force_key_frames "expr:gte(t,n_forced*5)" -movflags +faststart -c:a copy -f rtsp -rtsp_transport tcp rtsp://0.0.0.0:8554/stua

So basically adding -threads 1 stopped the decoding errors that appeared every few minutes, and -force_key_frames "expr:gte(t,n_forced*5)" made the stream start showing video within 5 seconds vs within 20 seconds.

Update, do not use the force_key_frames options. It completely messes up the vedio when there is motion. So now I'm back to having the stream be slow to start, but working with frigate.

@brujoand
Copy link
Author

brujoand commented Nov 28, 2021

So now both streams are working and looking perfect in VLC, and Frigate seems happy. But whenever events are recorded they look terrible. There is tearing, artifacts and wherever there is movement the screen gets a green layer. But theres no error in the frigate logs though and the debug/detect stream looks the same as in VLC. Any ideas?

Edit: So setting the bitrate -b:v 6M -maxrate 6M fixed the artifacts and discoloring of the recorded video in Frigate. But there's still somethig very wrong with the recording. It looks as though frames are being displayed slightly out of order so that moving objects seems to vibrate or jitter.

@brujoand
Copy link
Author

brujoand commented Nov 28, 2021

So I logged into the frigate container and ran the command to save the 'record' stream to disk manually. Then I get a bunch of output that is was not included from the frigate logs. Basically a never ending stream of:

[segment @ 0x563627270d00] Non-monotonous DTS in output stream 0:0; previous: 256282, current: 256282; changing to 256283. This may result in incorrecttimestamps in the output file.
[segment @ 0x563627270d00] Non-monotonous DTS in output stream 0:0; previous: 257419, current: 257419; changing to 257420. This may result in incorrecttimestamps in the output file.
[segment @ 0x563627270d00] Non-monotonous DTS in output stream 0:0; previous: 258529, current: 258529; changing to 258530. This may result in incorrecttimestamps in the output file.
[segment @ 0x563627270d00] Non-monotonous DTS in output stream 0:0; previous: 259869, current: 259869; changing to 259870. This may result in incorrecttimestamps in the output file.
[segment @ 0x563627270d00] Non-monotonous DTS in output stream 0:0; previous: 261993, current: 261993; changing to 261994. This may result in incorrecttimestamps in the output file.

So I'm guessing my transcoder stream isn't providing a monotonous DTS, which trips up the stream. What I don't get is why VLC can play the stream just fine with no DTS messegas. The only output I can find from VLC is this:

main debug: picture might be displayed late (missing 5 ms)
main warning: picture is too late to be displayed (missing 32 ms)
main debug: picture might be displayed late (missing 19 ms)
main debug: picture might be displayed late (missing 21 ms)
main warning: playback too early (-43656): down-sampling
main debug: picture might be displayed late (missing 2 ms)
main debug: picture might be displayed late (missing 5 ms)
main warning: playback too late (64689): up-sampling
main warning: playback too early (-40068): down-sampling

There are just a few of these in VLC, but a storm in ffmpeg in the Frigate container with the DTS message. I guess if a few frames are in the wrong place, then VLC is just ignoring them, but ffmpeg is trying to fit them into the video.

Update: What really bugs me about this is that if I just download the stream like so:
ffmpeg -i rtsp://10.30.0.5:8554/stua -c:v copy test.mp4
I can play it just fine, with no stuttering. But the segment creation that Frigate does with ffmpeg somehow get's all jittery.

@ozett
Copy link

ozett commented Nov 29, 2021

have you check if h264 encoding in hardware is fully supported?
or fast enough?

https://wiki.libav.org/Hardware/vaapi

i am no expert, but may it helps to find the right answer.
i had some strong fiddling to find out that my hardware does only support h264 enc/dec, but no h265 support. thats were ffmpeg-sw falls back to decoding defaults and maybe performance is an issue?

@brujoand
Copy link
Author

Yeah my Xeon E-2224G (Intel® UHD Graphics P630) fully supports decode and encode of both formats in hardware. And the interesting thing here is that the video plays just fine in VLC. That is VLC still prints messages about some frames being too soon, and some being too late. Playing the video in quicktime however will reveal the choppyness, same as in my browser. It's really weird tbh.

@blakeblackshear
Copy link
Owner

But the segment creation that Frigate does with ffmpeg somehow get's all jittery.

Are you certain it's the segment creation? Try adding the segment options to that simple command to check.

@ozett
Copy link

ozett commented Nov 29, 2021

Yeah my Xeon E-2224G (Intel® UHD Graphics P630) fully supports decode and encode of both formats in hardware.

only from my experiments with ffmpeg: its mostly invisivle if ffmpeg falls back to software-defaults. even if hardware could do, ffmpeg must use it. and hard to find a clue, if ffmpeg uses the configured hw-support. instead fallen silently back to softwaer.

but your problem is maybe something else...

@brujoand
Copy link
Author

Are you certain it's the segment creation? Try adding the segment options to that simple command to check.

Actually no, that was wrong. It's the option -use_wallclock_as_timestamps 1 which creates the stuttering. If I remove that from the frigate ffmpeg command, the video is finally stored properly. I'm not completly sure why, but I think my stream is sending frames a bit out of order, and that options forces them to be set in the order the arrived. Why I'm sending frames out of order though I have no idea.

@blakeblackshear
Copy link
Owner

Does that solve your issue? It's ok to remove that parameter if needed.

@brujoand
Copy link
Author

Yeah, I don't want to dig any deeper into this rabbit hole than I have to, and now it works. Thanks, let's close this 👍

@epicurean70
Copy link

Hi brujoand,
can you share your final stable ffmpeg settings?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants