OMX_EmptyThisBuffer hangs when encoding 1280 x 720 video via ffmpeg

[jasaw]

I'm running motion software with ffmpeg h264_omx encoder and experiencing OMX_EmptyThisBuffer function hanging when encoding 1280 x 720 resolution video. It works flawlessly at 800 x 600 resolution, 1920 x 1080 appears to work too but haven't tested it much.

I've added start_debug=1 to config.txt, and sudo vcdbg log assert gave me nothing.
My /etc/modprobe.d/bcm2835-v4l2.conf: options bcm2835-v4l2 max_video_width=1920 max_video_height=1080

How do I debug this further?

More information about the versions tested here.

[jasaw]

ffmpeg libavcodec/omx.c has input_zerocopy feature for optimization (in omx_encode_frame function). It appears that whenever the zerocopy condition is satisfied (contiguous planes and stride alignment), the call to OMX_EmptyThisBuffer hangs.

Why would OMX_EmptyThisBuffer hang when it is given a different buffer?

[6by9]

Nice detective work.
800x600 - 600 is not multiple of 16 for nSliceHeight (608 is), so needs a copy.
1920x1080 - 1080 is again not multiple of 16 for nSliceHeight (1088 is), so needs a copy.

Can I check my understanding? It runs for some time at 1280x720, but then hangs? Or does it hang on the first frame?
If the former then I'd guess FFMPEG may be submitting more buffers than nBufferCount is set to - if you can add logging to that then you may be able to confirm that.
The GPU side will have allocated nBufferCount buffers on the GPU side to accept the submitted data. Sending more buffers than that will therefore mean that there isn't a buffer available to accept it. Then again for that to be the case FFMPEG will have had to magic up some extra buffer headers too.

Have you tried adding start_debug=1 to config.txt, and checking sudo vcdbg log assert when it has occurred? That logs GPU side asserts (there are about 2 always present on boot, but otherwise the log should be clear).

[jasaw]

@6by9 Thank you very much for your response. It makes perfect sense now.

I think this is what happened:

• when motion uses MMAL interface to get an image from the camera, it copies the image into the ARM memory space.
• eventually, motion hands the image over to ffmpeg to encode, with h264_omx being the encoder.
• ffmpeg enables zerocopy by default on Raspberry Pi, so hands over the image in ARM memory space, which is causing the problem.

If I want motion to support zerocopy feature, what's the correct way of doing it?

Currently, this is what motion does:

• motion calls mmal_queue_wait to get a camera buffer.
• motion calls mmal_buffer_header_mem_lock then memcpy to a malloc'ed buffer, then unlocks with mmal_buffer_header_mem_unlock.
• motion calls mmal_buffer_header_release to release the camera buffer.

To support zerocopy, what does motion need to do?

• enable zerocopy with mmal_port_parameter_set_boolean(camera_port, MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE)
• use mmal_port_pool_create to create pool.
• call mmal_queue_wait to get a camera buffer.
• lock shared memory with mmal_buffer_header_mem_lock.
• how should motion deal with the camera image?
• unlock shared memory with mmal_buffer_header_mem_unlock.
• when can motion release camera buffer with mmal_buffer_header_release?

[6by9]

If I understand you correctly, using MMAL_PARAMETER_ZERO_COPY is going to be the key here.
I suspect it is the same as raspberrypi/userland#386 which I've only half got to the bottom of.

For reasons I don't fully understand the kernel has some magic flags against each allocation mapping. If a range of pages is mapped into memory using remap_pfn_range or a few other calls then it sets the VM_IO and/or VM_PFNMAP flags. On trying to call get_user_pages to map the userspace pointer back into a list of pages, those flags are checked and -EFAULT returned if they are set.

Using zero copy means that the gpu_mem allocation is mapped into the kernel and ends up with one of those flags set. That means VCHIQ as part of IL, MMAL, or DispmanX gets the failure on get_user_pages.
You could try picking 6by9/linux@5dc3b62 from my https://github.com/6by9/linux/tree/unicam_wip tree as I hit the same issue when trying to pass a CMA mapped address back into VCHIQ. I think I tried it with userland#386 and it didn't help as it was still misinterpreting the mapping, but that could be my memory playing tricks on me.

As to what needs to be done:

• yes, mmal_port_parameter_set_boolean(camera_port, MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE)
• yes, mmal_port_pool_create(camera_port, count, size)
• mmal_port_enable(camera_port, callback), where callback is going to do something with the returned buffer.
• either in the callback, or from a different thread picking up on a queue from the callback, take the buffer and use the data pointed to by buf->data within motion.
• when done, send the buffer back to camera_port if it is still enabled. You don't need to return it to the pool with mmal_buffer_header_release, but that can be a convinient way of handling the buffers by either just looping pulling all buffers out of the pool and sending to the component, or by adding an "on-release" callback to the pool to automatically send the buffer to the component. (Returning it to the pool if the port is disabled is a useful way to stash the buffer so you don't lose it forever)

Don't worry about mmal_buffer_header_mem_lock and mmal_buffer_header_mem_unlock. If you check the code for them they have no effect outside of Videocore. https://github.com/raspberrypi/userland/blob/master/interface/mmal/core/mmal_buffer.c#L157. The MMAL framework has automatically dealt with swapping buffer->data to being a userspace pointer (https://github.com/raspberrypi/userland/blob/master/interface/mmal/vc/mmal_vc_api.c#L511 and https://github.com/raspberrypi/userland/blob/master/interface/mmal/vc/mmal_vc_api.c#L637)

[andreluisos]

I have this very same issue. Ffmpeg freezes after 1-6 minutes of constant capture and encoding with h264_omx, deppending on the resolution, but only if it's 720p or higher.

RPI 3b+
ffmpeg version 3.2.10-1~deb9u1+rpt2

[6by9]

@andreluisos I don't believe you have the same issue.
This one is that it fails to even encode one frame, whilst for you it runs for several minutes.

[tmm1]

I can reproduce this OMX_EmptyThisBuffer hang easily in ffmpeg by combining vf_scale_v4l2m2m with h264_omx.

(gdb) bt
#0  0x76f1d088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1,
    futex_word=0xb8abe4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xb8abe4, abstime=0x0) at sem_waitcommon.c:115
#2  0x76f1d1f4 in __new_sem_wait_slow (sem=0xb8abe4, abstime=0x0) at sem_waitcommon.c:282
#3  0x6ea908bc in ilcs_execute_function_ex () from /opt/vc/lib/libopenmaxil.so
#4  0x6ea918a4 in ilcs_pass_buffer () from /opt/vc/lib/libopenmaxil.so
#5  0x001f2f0c in omx_encode_frame (avctx=0xb48040, pkt=0xbe89d0, frame=0xbe8b20, got_packet=0x7effe214)
    at libavcodec/omx.c:822
#6  0x0013d29e in avcodec_encode_video2 (avctx=avctx@entry=0xb48040, avpkt=0xbe89d0,
    frame=frame@entry=0xbe8b20, got_packet_ptr=got_packet_ptr@entry=0x7effe214)
    at libavcodec/encode.c:296
#7  0x0013d458 in do_encode (avctx=0xb48040, frame=0xbe8b20, got_packet=0x7effe214)
    at libavcodec/encode.c:365
#8  0x0013d524 in avcodec_send_frame (avctx=avctx@entry=0xb48040, frame=frame@entry=0xbe8b20)
    at libavcodec/encode.c:414
#9  0x000543b4 in do_video_out (of=of@entry=0xb489b0, ost=ost@entry=0xb81d50,
    next_picture=next_picture@entry=0xbe8b20, sync_ipts=63063.000007629395) at fftools/ffmpeg.c:1289
#10 0x00057896 in reap_filters (flush=0, flush@entry=12066128) at fftools/ffmpeg.c:1506
#11 0x0005b196 in transcode_step () at fftools/ffmpeg.c:4663
#12 transcode () at fftools/ffmpeg.c:4707
#13 0x00042124 in main (argc=<optimized out>, argv=<optimized out>) at fftools/ffmpeg.c:4909

With the fix for https://trac.ffmpeg.org/ticket/6586 in http://ffmpeg.org/pipermail/ffmpeg-devel/2019-August/248784.html, you can now work-around the issue by passing -zerocopy 0 to the ffmpeg cli.