Raspberry Pi 4 ffmpeg expectations?

[colemickens]

Hi! I am trying to calibrate expectations. We have an RTSP feed feeding into Frigate that is an h264 feed, and we have the Coral running as a detector. I've seen the CPU decrease as we activated the Coral, and I've seen ffmpeg correctly using h264_v4l2m2m but I'm also still seeing this single camera running ffmpeg in such a way that eats 25-50% of the CPU while it's going.

What should my expectations be? I've seen some things that indicate that the colorspace conversion to RGB for display can be the intensive part. Is it possible to skip that?

[blakeblackshear]

What is the resolution and frame rate of the feed you are decoding with ffmpeg?

The color space is only converted when necessary to run object detection. It's not possible to avoid, but I don't think that's really the issue here either.

[colemickens]

By the way, when testing with the latest stable, I'm getting much, much better results. I don't have any exact stats, but I've got the rpi4 handling 7x 700x400 feeds right now, with detection and clip recording all working pretty painlessly. Needless to say this is within or exceeding my original expectations.

[colemickens]

(I did bump gpu_mem by manually putting the SD Card in a Linux machine, mounting the firmware part, editting config.txt, etc.) Before I bumped it, I saw ffmpeg spitting out weird v4l2m2m errors after adding more than a couple of streams.

[blackjack75]

Out of curiosity have you been able to use hardware acceleration for decoding more than 6 feeds ? I see you mentioned having 7.

I have a Raspberry 4, overclocked and stable at 2137 Mhz (with an Argon case ventilating heavily!). gpu_mem set to 512.

I manager to run 6 feeds (source 960x480 at 10 fps, from an IntelBras DVR) with hw_accel on .

As soon as I add a 7th feed I start getting green images and a lot of decoding errors in the frigate logs.

Interesting point this doesn't seem to be about overloading the CPU. I can add 2 more streams and reach a total 8 seemingly stable as long as the last 2 feeds have hardware decoding disabled. Of course these two last feed hit the cpu much more brutally.

I run a 32 bit RaspberryOS BTW. Plan to reinstall a dedicated RPI4 with 64 bit soon. Not sure if this has a significant effect.

[blackjack75]

Just adding the kind of errors I get in the logs when enabling 8 cameras with acceleration instead of 6. I tried increasing gpu_mem to the safe limit (896M) but it didn't change anything.

frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : [h264_mmal @ 0x137e8a0] MMAL error 3 when sending output buffer.
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : Error while decoding stream #0:0: Unknown error occurred
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : [h264_mmal @ 0x137e8a0] MMAL error 3 when sending output buffer.
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : Error while decoding stream #0:0: Unknown error occurred
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : [h264_mmal @ 0x137e8a0] MMAL error 3 when sending output buffer.
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : Error while decoding stream #0:0: Unknown error occurred
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : [h264_mmal @ 0x137e8a0] MMAL error 3 when sending output buffer.
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : Error while decoding stream #0:0: Unknown error occurred
frigate    | [2021-11-05 21:14:38] ffmpeg.smart_cam_08.detect     ERROR   : [h264_mmal @ 0x137e8a0] MMAL error 3 when sending output buffer.```

[blackjack75]

One more interesting experience on the same issue: setting all feeds down to the minimum resolution (352x240, down from 960x480).

Surprisingly it does NOT solve the issue. CPU load goes down a chunk of course but the issue with accelerated feeds still happen as soon as I go over 6 feeds. Dividing the total number of pixels to be treated doesn't change it. Still green and decoding errors.

So it's really something about the number of ffmpeg processes doing decoding ?

[blackjack75]

So I finally isolated the command that failed. I launch 4 cams via the config file and and manually run the following command three times. It works twice, and then on the third time (which would be the 7th feed) I get this error below.

ffmpeg -hide_banner -loglevel warning -c:v h264_mmal -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i "rtsp://santiago:SOMEPASS@192.168.0.33:554/cam/realmonitor?channel=1&subtype=1" -r 5 -s 960x480 -f rawvideo -pix_fmt yuv420p pipe: > /dev/null

[h264_mmal @ 0x16c7480] MMAL error 2 on control port
mmal: mmal_port_event_get: vc.ril.video_decode(1:0) port 0x14929c0, no event buffer left for ERRO
mmalipc: mmal_vc_handle_event_msg: no event buffer available to receive event (2)
mmal: mmal_port_event_get: vc.ril.video_decode(1:0) port 0x14929c0, no event buffer left for ERRO
mmalipc: mmal_vc_handle_event_msg: no event buffer available to receive event (2)
mmal: mmal_port_event_get: vc.ril.video_decode(1:0) port 0x14929c0, no event buffer left for ERRO
mmalipc: mmal_vc_handle_event_msg: no event buffer available to receive event (2)
mmal: mmal_port_event_get: vc.ril.video_decode(1:0) port 0x14929c0, no event buffer left for ERRO
mmalipc: mmal_vc_handle_event_msg: no event buffer available to receive event (2)
(...)

Last message repeated 22 times
[h264_mmal @ 0x16c7480] Did not get output frame from MMAL.
Error while decoding stream #0:0: Unknown error occurred
[h264_mmal @ 0x16c7480] MMAL error 2 on control port