omx video decode api

[cleverca22]

I've been working on trying to get mplayer to run on the pi with full gpu decode, and I've hit a wall with the lack of documentation on the OMX interface

http://bpaste.net/show/151411/

I think I've properly adapted the hello_video.c code to remove the scheduler and accept data from player, but it just hangs at ilclient_get_input_buffer and never gets a buffer to put data into

is where any documentation on things like the ports in set_tunnel(&tunnel,video_decode,131,video_render,90); or how its supposed to be used, beyond hello_video?

[popcornmix]

The components and ports used are documented here:
https://github.com/raspberrypi/firmware/tree/master/documentation/ilcomponents
You will need to read this:
http://www.khronos.org/registry/omxil/specs/OpenMAX_IL_1_1_2_Specification.pdf

VLC does contain Pi specific openmax decode so you may want to look there for an example.

[cleverca22]

those docs look perfect!, now i just need to sit down and read 400+ pages

edit1:
i think the problem was either things like #define OMX, or how i copy/pasted ilclient.c into my file
its getting farther now that i'm linking against ilclient properly
edit2:
aha!, if i compile the demo hello_video without the proper makefile, it doesnt work, so one of those -D flags is key

[cleverca22]

ok, ive narrowed down the problem, ffmpeg is removing this header from the h264 data stream: http://pastebin.com/7LyarVYy

is there an alternative way to give that data to omx, or should i just recover the header and put it at the start of the data where it belongs?

[popcornmix]

The header bytes should be submitted to video decode with the OMX_BUFFERFLAG_CODECCONFIG flag. See:
https://github.com/popcornmix/omxplayer/blob/master/OMXVideo.cpp#L89

You can just prepend them to the first video frame at the start of decode, but submitting them with the right flag means they get reapplied after flushes etc.

[cleverca22]

the omx code you referenced is using ffmpeg's codec->extradata, which i have tried, but that header is much shorter and doesnt work under mplayer

ive tried to figure out if extradata is different under omxplayer, but it wont compile on any system i have, natively or cross

is there any documents on what the bytes in that header mean, so i can figure out what differs between the different copies i get?

edit1: got the native build on the pi started, but its going slow...
edit2: OMXPlayerSubtitles.cpp:172:22: error: ‘steady_clock’ is not a member of ‘std::chrono’
cant compile omxplayer, so i cant figure out why its extradata differs

[popcornmix]

The extradata bytes are codec specific. E.g. for h.264 they tend to contain SPS/PPS NAL units. You would need to refer to the h.264 spec for interpret them.

You need gcc 4.7 to compile omxplayer. See:
http://stackoverflow.com/questions/16518705/probleme-with-stdchrono

[cleverca22]

yeah, the prepare script in omxplayer installed 4.7 and changed the symlinks for it, so it should have worked

edit1: oh, its using sh and tried to use pushd, it failed to make the symlinks to force 4.7
edit2: its trying to use bash features with #!/bin/sh
https://github.com/popcornmix/omxplayer/blob/master/prepare-native-raspbian.sh#L43
edit3: OMXControl.cpp:7:23: fatal error: dbus/dbus.h: No such file or directory
and the prepare script didnt get dbus dev packages

[popcornmix]

@cleverca22
I believe the prepare script works with stock raspbian image.
If you have patches to make it work more generically, then a PR would be welcome.

[cleverca22]

i'll double check the omx issues once i'm done this mplayer patch
it looks like the symlink for gcc -> gcc-4.7 also broke the mplayer build, the .o files arent cross compatible

[cleverca22]

the extradata that omxplayer has is identical to what mplayer is giving me, i'll need to investigate the differences more to see why it works in omxplayer but not in mplayer

[cleverca22]

ilclient_enable_port_buffers(video_decode, 131, NULL, NULL, NULL);
out = ilclient_get_output_buffer(video_decode,131,1);
OMX_FillThisBuffer(ILC_GET_HANDLE(video_decode), out);

this this the right way to read raw frames out, and can you read them after giving it the slice data for just a single frame?

it seems to hang on get_output_buffer after a frame or 2, and only returns nFilledLen 0 before that
will it also return 0 and/or hang if the video data isnt valid?

[popcornmix]

It's probably worth starting with some working code and (carefully) adapting it to your use case.
OpenMAX is very much state based, and in general it is very easy to break code by moving it around.

I believe VLC has openmax decode similar to your requirements. I'd suggest looking there:
https://github.com/julianscheel/vlc-omx/tree/master/modules/codec/omxil

[cleverca22]

just to figure out how it works, i tried to modify hello_video to dump raw YUV420 frames, but that isnt working either

acording to hello_videocube, i should be using ilclient_set_fill_buffer_done_callback to detect when the output is ready

checking ilclient.c, i can see that the component must be created with ILCLIENT_ENABLE_OUTPUT_BUFFERS for that callback to even work

and i am calling ilclient_enable_port_buffers on port 131 to enable its output

the callback never gets ran, and after about 120 input buffers worth of data, the input buffers start blocking, and it just hangs

https://gist.github.com/cleverca22/7690326

edit1: after going over a bit of the VLC code, it appears to not be using il but just raw omx, OMX_UseBuffer appears to be passing an invalid pointer, (void*)1, as the buffer address, so i'm not sure how VLC even works, it doesnt have anything similar to ilclient_enable_port_buffers

[cleverca22]

found out one part of why things arent working the same
i modified omxplayer to dump its buffer to a file at every call to EmptyThisBuffer

then i ran hello_video on that, which just reads them and feeds the data over, it did the same thing as my patches to ffmpeg/mplayer (nothing at all)

there must be something special that omxplayer is doing that hello_video doesnt

[cleverca22]

is there any valid way to get 2 full frame output buffers from resize or video_decode?
i tried calling OMX_UseBuffer twice, and it errors, and checking the portdef, it has a nBufferCountActual of 1

is there any special flags to allow nBufferCountActual 2, or is it just not possible?

edit1: hmmm, the pdf section 3.1.1.2.1.1 says that i can use OMX_SetParameter to change the number of buffers, is that only to increase it up to the nBufferCountActual, or can i set it as high as i want?