Corrupted video with h264_v4l2m2m at Pi camera resolution

[Malvineous]

Describe the bug
Using H264 hardware encoding only works for certain resolutions. If you use one of the resolutions supported by the Pi camera, the image is corrupted.

To reproduce
This command creates a test image of the same resolution as one of the Pi camera settings. It then encodes it using the hardware H264 encoder, and then immediately decodes it again in software and displays it:

ffmpeg -f lavfi -i testsrc=size=1296x972:rate=30 -pix_fmt yuv420p -c:v h264_v4l2m2m -f avi - | ffplay -

Expected behaviour
I expected the image to be clear, as it is if you change the size parameter in the above command to a different value, such as 1920x1080.

Actual behaviour
The image is corrupted at 1296x972 and similar resolutions:

System

• Which model of Raspberry Pi? Pi4
• Which OS and version (cat /etc/rpi-issue)? Arch Linux ARM 64-bit
• Which firmware version (vcgencmd version)?

Jul 21 2021 16:21:46 
Copyright (c) 2012 Broadcom
version 6a796bb0062a6c75191c57cba1c13f9300076d02 (clean) (release) (start)

• Which kernel version (uname -a)? Linux 5.10.50-2-ARCH #1 SMP PREEMPT Mon Jul 19 16:00:34 UTC 2021 aarch64 GNU/Linux

Additional info
I was hoping to expand the video from a Pi camera vertically a little, and add a status line with the current time, then encode that as H264 but I found the image was corrupted. It seems it is corrupted at various resolutions around this, but strangely the larger resolution of 1920x1080 is fine.

[6by9]

I'm fairly certain this is FFmpeg getting image strides wrong, or not propagating them between components properly.

V4L2 advertises a format which has width, height, and bytesperline (aka stride or pitch).
bytesperline allows the start of each line to be aligned to a helpful multiple to make memory accesses more efficient, and should always be a multiple of 32 pixels.

When the format is passed from source to sink the bytesperline needs to be acceptable to both ends.
1296 is not divisible by 32 (it's 40.5*32), but 1920 is.

[Malvineous]

Interesting. I tried different widths, and 1280 (40*32) works, 1344 (42*32) works, but 1321 (41*32) does not work. Some more experimentation suggests widths divisible by 64 are the only ones that work. The widths that are broken (including those that are a multiple of 32) also show this message repeatedly in the logs I didn't notice before:

[h264_v4l2m2m @ 0x556a9d8e00] capture driver encode error

The decoding step (software decoder processing the output of the hardware encoder) also produces a number of repeating error messages when processing the resulting video, apologies for not noticing these earlier:

[h264 @ 0x7f84002150] missing picture in access unit with size 47
[h264 @ 0x7f84002150] no frame!
[h264 @ 0x7f84002150] missing picture in access unit with size 524288
[h264 @ 0x7f84002150] No start code is found.
[h264 @ 0x7f84002150] Error splitting the input into NAL units.

Would a stride error result in these decoder messages reporting a corrupted stream? Or would it result in correctly encoded H264 data just having the wrong picture?

Is it significant that the lines covering the intensity (black and white) seem intact, while it only appears to be the colour information that has the wrong stride?

[6by9]

Another thread has reminded me that we need to implement the same fix as raspberrypi/linux#4419 for bcm2835-codec (decode, encode and simple ISP). Decode uses the ISP to format convert the final frames. Encode uses them to format convert the source frames. ISP uses the ISP. They all therefore have the same restriction on the stride being a multiple of 32 for yuv420p and yvu420p.

[dustinkerstein]

Another thread has reminded me that we need to implement the same fix as raspberrypi/linux#4419 for bcm2835-codec (decode, encode and simple ISP). Decode uses the ISP to format convert the final frames. Encode uses them to format convert the source frames. ISP uses the ISP. They all therefore have the same restriction on the stride being a multiple of 32 for yuv420p and yvu420p.

Does this mean that in order to use h264_v4l2m2m encoder, the output width and height will always need to be divisible by 32? I just ran into this issue and was scratching my head for a while. Do you think there will be a way to lift this limitation?

Also, depending on the source (raw yuv420p vs. h264) I sometimes get either scrambled green output, or a segmentation fault with no output.

Segmentation fault (using this 1080p30 Big Buck Bunny test video):
ffmpeg -y -i input.mp4 -filter:v crop=1280:720 -c:v h264_v4l2m2m out.mp4

Scrambled green output:
ffmpeg -y -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 30 -i input.yuv -filter:v crop=1280:720 -c:v h264_v4l2m2m -pix_fmt yuv420p out.mp4

Correct output:
ffmpeg -y -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 30 -i input.yuv -filter:v crop=1920:1080 -c:v h264_v4l2m2m -pix_fmt yuv420p out.mp4

FYI, I see the same behavior on the current ffmpeg in the Bullseye repos, and with compiling the latest ffmpeg.

[dustinkerstein]

Quick update - Just repeated the above tests on Buster with h264_omx instead of h264_v4l2m2m and it encodes correctly with no glitches / color overlay.

[Malvineous]

That's interesting, that makes me wonder whether the problem is V4L2 reporting the wrong bytesperline value?

[dustinkerstein]

Not sure if it's helpful to note, but the RidgeRun(used by Nvidia)/Arducam IMX477 driver exposes a 4032x3040 mode rather than 4056x3040. And I know from experience that their v4l2 h264/265 encoders work at that 4032x3040 resolution.