Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MMAL video decoder sometimes outputs corrupted timestamps #456

Closed
ghost opened this issue Aug 11, 2015 · 38 comments
Closed

MMAL video decoder sometimes outputs corrupted timestamps #456

ghost opened this issue Aug 11, 2015 · 38 comments

Comments

@ghost
Copy link

ghost commented Aug 11, 2015

I'm not really sure what's happening here. But it almost looks like MMAL is outputting corrupted timestamps every 10 seconds with a specific sample I have.

Here are the first 15 seconds or so of the sample: https://0x0.st/sV_.mkv

With ffmpeg -c:v h264_mmal -i sample.mkv -vf showinfo -f null - I get this at around the 10 second mark (use FFmpeg git master as of now; older versions have a bug):

[Parsed_showinfo_0 @ 0x1d5f0c0] n: 248 pts:  10344 pts_time:10.344  pos:  3503692 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:1B950DC2 plane_checksum:[78DD99FA 0A659D1D 47F3D68D] mean:[99 125 133] stdev:[40.8 10.0 10.3]
[Parsed_showinfo_0 @ 0x1d5f0c0] n: 249 pts:  10385 pts_time:10.385  pos:  3520414 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:A53CA625 plane_checksum:[46F0DA3F 34EA651D 581366BA] mean:[99 125 133] stdev:[41.3 9.9 10.2]
[Parsed_showinfo_0 @ 0x1d5f0c0] n: 250 pts:  52093 pts_time:52.093  pos:  3549459 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:1D6286E5 plane_checksum:[E445B642 80B84E4F 534B8245] mean:[98 124 133] stdev:[41.7 9.8 10.1]
[Parsed_showinfo_0 @ 0x1d5f0c0] n: 251 pts:  10469 pts_time:10.469  pos:  3565022 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:846BF201 plane_checksum:[7FF96E8C C144BFE4 E152C382] mean:[98 124 133] stdev:[42.1 9.8 10.0]
[null @ 0x1db8b30] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1249 >= 251

Frame number 250 outputs an odd timestamp. Frame number 251 is back to normal.

It happens after the next 250 frames too:

[Parsed_showinfo_0 @ 0x1b178b0] n: 499 pts:  20812 pts_time:20.812  pos:  7366248 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:92C985CC plane_checksum:[5B9282A8 8780C19F 12244176] mean:[46 135 122] stdev:[47.5 9.2 10.2]
[null @ 0x1b6b830] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1249 >= 499                                             
[Parsed_showinfo_0 @ 0x1b178b0] n: 500 pts:  62520 pts_time:62.52   pos:  7383549 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:698E2BFF plane_checksum:[7911A759 6150BC3F DEEEC849] mean:[46 135 122] stdev:[47.5 9.2 10.1]       
[Parsed_showinfo_0 @ 0x1b178b0] n: 501 pts:  20896 pts_time:20.896  pos:  7386065 fmt:yuv420p sar:1/1 s:1920x800 i:P iskey:1 type:? checksum:6AE589CA plane_checksum:[ACF45616 2EB42ABA 1B5308FA] mean:[46 135 122] stdev:[47.4 9.1 10.1] 

(not included in the sample, but I can upload the full one if needed)

But the packet timestamp seems to be fine:

[PACKET]
codec_type=video
stream_index=1
pts=10427
pts_time=10.427000
dts=10344
dts_time=10.344000
duration=41
duration_time=0.041000
convergence_duration=N/A
convergence_duration_time=N/A
size=50353
pos=3270843
flags=K
[/PACKET]

(from ffprobe sample.mkv -show_packets)

The FFmpeg h264 software decoder has no issues here, it outputs the frame like this:

[FRAME]
media_type=video
stream_index=1
key_frame=1
pkt_pts=10427
pkt_pts_time=10.427000
pkt_dts=10427
pkt_dts_time=10.427000
best_effort_timestamp=10427
best_effort_timestamp_time=10.427000
pkt_duration=41
pkt_duration_time=0.041000
pkt_pos=3270843
pkt_size=50353
width=1920
height=800
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=I
coded_picture_number=250
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
[/FRAME]

(from ffprobe -show_frames -select_streams 1 sample.mkv)

Is it possible that MMAL is doing something wrong here?

(Side note: ffprobe is a useful tool, but you can't force the video decoder yet - so it can't be used with the mmal wrapper.)

@popcornmix
Copy link
Contributor

What does vcgencmd version report?
There's been some fixes to timestamp logic and I'd like to know if you have a problem with old firmware or new firmware.

@popcornmix
Copy link
Contributor

Looking at the file with kodi using mmal:
I submit:

11:34:23 1050.473999 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0080 18698 /0      dts:0.000 pts:10.052 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.514282 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0258 5052  /0      dts:0.000 pts:10.010 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.555908 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0430 13972 /0      dts:0.000 pts:10.093 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.597656 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0608 14063 /0      dts:0.000 pts:10.135 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.641602 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae07e0 29325 /0      dts:0.000 pts:10.302 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:23 1050.681274 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae09b8 13531 /0      dts:0.000 pts:10.219 flags:10000004 ready_queue(2) demux_queue(0) space(1556480)
11:34:23 1050.722900 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0b90 5240  /0      dts:0.000 pts:10.177 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.765015 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0d68 7168  /0      dts:0.000 pts:10.260 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:23 1050.807251 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0f40 19819 /0      dts:0.000 pts:10.385 flags:10000004 ready_queue(2) demux_queue(0) space(1556480)
11:34:23 1050.848022 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1118 5394  /0      dts:0.000 pts:10.344 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.890015 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae12f0 50353 /0      dts:0.000 pts:10.427 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1050.931763 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae14c8 20461 /0      dts:0.000 pts:10.511 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:23 1050.974854 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae16a0 5308  /0      dts:0.000 pts:10.469 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1051.015137 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1878 17340 /0      dts:0.000 pts:10.552 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:23 1051.056885 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1a50 31555 /0      dts:0.000 pts:10.719 flags:10000004 ready_queue(2) demux_queue(0) space(1556480)
11:34:23 1051.098511 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1c28 13965 /0      dts:0.000 pts:10.636 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1051.142334 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1e00 6244  /0      dts:0.000 pts:10.594 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:23 1051.182007 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae1fd8 11037 /0      dts:0.000 pts:10.677 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1051.223755 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae21b0 28202 /0      dts:0.000 pts:10.844 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:23 1051.265869 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3adfea8 11567 /0      dts:0.000 pts:10.802 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)
11:34:24 1051.308105 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0080 8090  /0      dts:0.000 pts:10.761 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:24 1051.349243 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0258 16716 /0      dts:0.000 pts:10.886 flags:10000004 ready_queue(2) demux_queue(0) space(1556480)
11:34:24 1051.390747 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0430 14711 /0      dts:0.000 pts:10.928 flags:10000004 ready_queue(3) demux_queue(0) space(1556480)
11:34:24 1051.432373 T:1608512544   DEBUG: CMMALVideo::Decode - 0x3ae0608 15556 /0      dts:0.000 pts:10.969 flags:10000004 ready_queue(4) demux_queue(0) space(1556480)

and I get back:

11:34:23 1051.173340 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adea90 (0x3c537a0) buffer_size(128) dts:-9223372036854.775 pts:10.010 flags:10000004:0
11:34:23 1051.173462 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adee40 (0x3adc9e8) buffer_size(128) dts:-9223372036854.775 pts:10.052 flags:10000004:0
11:34:23 1051.214600 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adec68 (0x3f3f270) buffer_size(128) dts:-9223372036854.775 pts:10.093 flags:10000004:0
11:34:23 1051.256226 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf018 (0x334da00) buffer_size(128) dts:-9223372036854.775 pts:10.135 flags:10000004:0
11:34:24 1051.382202 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf1f0 (0x3f2a750) buffer_size(128) dts:-9223372036854.775 pts:10.177 flags:10000004:0
11:34:24 1051.382446 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf3c8 (0x3376fb0) buffer_size(128) dts:-9223372036854.775 pts:10.219 flags:10000004:0
11:34:24 1051.423462 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf5a0 (0x4103300) buffer_size(128) dts:-9223372036854.775 pts:10.260 flags:10000004:0
11:34:24 1051.423706 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf778 (0x3376e80) buffer_size(128) dts:-9223372036854.775 pts:10.302 flags:10000004:0
11:34:24 1051.507202 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf950 (0x3a20468) buffer_size(128) dts:-9223372036854.775 pts:10.344 flags:10000004:0
11:34:24 1051.507446 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adfb28 (0x30f11a8) buffer_size(128) dts:-9223372036854.775 pts:10.385 flags:10000004:0
11:34:24 1051.549316 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adea90 (0x3ad6358) buffer_size(128) dts:-9223372036854.775 pts:10.427 flags:4:0
11:34:24 1051.632446 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adee40 (0x3eb4060) buffer_size(128) dts:-9223372036854.775 pts:10.469 flags:10000004:0
11:34:24 1051.632690 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adec68 (0x3a8fa10) buffer_size(128) dts:-9223372036854.775 pts:10.511 flags:10000004:0
11:34:24 1051.673950 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf018 (0x3abfd00) buffer_size(128) dts:-9223372036854.775 pts:10.552 flags:10000004:0
11:34:24 1051.799194 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf1f0 (0x39aa878) buffer_size(128) dts:-9223372036854.775 pts:10.594 flags:10000004:0
11:34:24 1051.799438 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf3c8 (0x3a8f040) buffer_size(128) dts:-9223372036854.775 pts:10.636 flags:10000004:0
11:34:24 1051.841064 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf5a0 (0x68548968) buffer_size(128) dts:-9223372036854.775 pts:10.677 flags:10000004:0
11:34:24 1051.841309 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf778 (0x69a89908) buffer_size(128) dts:-9223372036854.775 pts:10.719 flags:10000004:0
11:34:24 1051.967041 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf950 (0x685d1ad0) buffer_size(128) dts:-9223372036854.775 pts:10.761 flags:10000004:0
11:34:24 1051.967407 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adfb28 (0x620009e8) buffer_size(128) dts:-9223372036854.775 pts:10.802 flags:10000004:0
11:34:24 1051.967651 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adea90 (0x69a3ef50) buffer_size(128) dts:-9223372036854.775 pts:10.844 flags:10000004:0
11:34:24 1052.007568 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adee40 (0x685b66c0) buffer_size(128) dts:-9223372036854.775 pts:10.886 flags:10000004:0
11:34:24 1052.049316 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adec68 (0x620ef7f0) buffer_size(128) dts:-9223372036854.775 pts:10.928 flags:10000004:0
11:34:24 1052.091187 T:1625289760   DEBUG: CMMALVideo::dec_output_port_cb - 0x3adf018 (0x69a514b8) buffer_size(128) dts:-9223372036854.775 pts:10.969 flags:10000004:0

So it looks okay here. There is something different about 10.427. It is an IDR frame and it comes back from the codec with timestamp unknown. It's a bit beyond my knowledge to say why, but this normally occurs when there are multiple frames in a packet (e.g. the vfw/avi packed bitstream hack).

However MMAL on Kodi sets MMAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS so the unknown timestamp gets set to the correct value. Without that I'd expect you to see MMAL_TIME_UNKNOWN as the pts value.

This assumes you are using latest rpi-update firmware. There were timestamp related bugs in earlier versions.

@ghost
Copy link
Author

ghost commented Aug 17, 2015

I haven't updated the firmware quite in a while, so maybe that's the problem and it's fixed now.

Currently I can't access my RPI though. (Both LEDs are lit without blinking, internet says it's a SD card problem, but both cards I tried work with the PC card reader.)

@popcornmix
Copy link
Contributor

Do you see rainbow splash screen on boot?
Basically some of the files on boot partition are not being read correctly.
Copy bootcode.bin, start.elf, fixup.dat from https://github.com/Hexxeh/rpi-firmware and you should at least get the rainbow splash (and possibly a complete boot).

@ghost
Copy link
Author

ghost commented Aug 17, 2015

No, the screen simply remained off all the time. Somehow I got it to start again (tried various combinations of disconnecting power and reinserting the SD card - maybe some physical issue).

The firmware was old, but after updating, I got the same behavior. The firmware version is now:

$ vcgencmd version
Aug 15 2015 17:52:41 
Copyright (c) 2012 Broadcom
version b22c2fd27dc8091883457064d8ebc3ac7c995309 (clean) (release)

I've also tried:

mmal_port_parameter_set_boolean(decoder->output[0],  MAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS, 1)

But it didn't seem to help.

Note that the code does handle MMAL_TIME_UNKNOWN. It's converted to/from ffmpeg's sentinel value, AV_NOPTS_VALUE. Though in this case, no input or output timestamps should have been MMAL_TIME_UNKNOWN.

@popcornmix
Copy link
Contributor

MMAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS does default to enabled, so you'd have to set it to 0 to disable.

Are you setting the framerate of the MMAL decoder?
popcornmix/xbmc@1fbde11

Having a bad value in there may cause unwanted effects with timestamp interpolation.

@ghost
Copy link
Author

ghost commented Aug 17, 2015

Are you setting the framerate of the MMAL decoder?

It's set to a dummy value.

@popcornmix
Copy link
Contributor

It's set to a dummy value.

What value? Does it behave better if not set or set to correct value?

@ghost
Copy link
Author

ghost commented Aug 17, 2015

It's 24000/1001. I can test changing it tomorrow; I'll also test whether disabling timestamp interpolation returns MMAL_TIME_UNKNOWN.

I've been wondering what the framerate field is about, since a decoder doesn't need it in theory. I'm also not sure what MMAL would do with the timestamps - since MMAL doesn't know the unit (and libavcodec also doesn't), they have any real meaning and are opaque.

@popcornmix
Copy link
Contributor

There are a few cases when timestamps are used by video_decode:
interpolating missing (unknown) timestamps
image_fx component with double rate deinterlace needs to create timestamps for extra frames
undoing the vfw/avi packed bitstream hack needs to know the timestamps
DTS timestamps

DTS timestamps are a problem. If your container has DTS but not PTS timestamps and includes B frames, then passing them through decoder will reorder the timestamps in a way you probably can't make use of (without parsing the video stream sufficiently to determine what reordering has occurred).

If you set the OMX_BUFFERFLAG_TIME_IS_DTS flag, then videodecode will try to associate unreordered timestamps with the frames that come back. However this is only possible in certain conditions (e.g. number of encoded frames matching number of decoded frames).

@ghost
Copy link
Author

ghost commented Aug 18, 2015

Changing the framerate passed to MMAL while interpolation is enabled doesn't change the bogus timestamp that is output..

If I disable timestamp interpolation, the MMAL decoder actually returns MMAL_TIME_UNKNOWN. Unfortunately, some FFmpeg generic code sets the DTS from the input packet, which is not useful, because the MMAL decoder is asynchronous. This lets higher level code determine a broken timestamp. (I missed this in my earlier tests.) I might be able to workaround this problem.

@popcornmix
Copy link
Contributor

If you can provide an example app that produces an unexpected interpolated timestamp for this frame I could debug it. (Can just be a binary - I don't need to debug the arm side).

@ghost
Copy link
Author

ghost commented Aug 18, 2015

Would a statically built mpv binary do?

@ghost
Copy link
Author

ghost commented Aug 18, 2015

(By which I mean it would only link to a few libs installed on Raspbian by default.)

@ghost
Copy link
Author

ghost commented Aug 18, 2015

Anyway, here's a mpv binary compiled under Raspbian: https://0x0.st/s6U.exe (the exe suffix was forced by the site)

Rename to mpv, chmod, start playback with ./mpv file.mkv. After 10 seconds with the given file it will complain about a timestamp oddity, and then sit around for a while doing nothing (because it's being stupid and thinks it has to show the current frame for a longer while - it doesn't actually freeze). The timestamp you'll see on the decoder level will seem odd to you, because they're IEEE doubles reinterpret-casted to int64. The behavior with the FFmpeg tool, which uses integer timestamps, is similar though, just that the resulting bogus timestamp has a different value.

@popcornmix
Copy link
Contributor

The timestamps I see are suspicious:

nTimeStamp = 0
nTimeStamp = 4595184829392702407 (0x3fc5604189374bc7)
nTimeStamp = 4590645200968312947 (0x3fb53f7ced916873)
nTimeStamp = 4586213658934980379 (0x3fa5810624dd2f1b)
nTimeStamp = 4593671619917905920 (0x3fc0000000000000)

I assume this is what you mean by "IEEE doubles reinterpret-casted to int64".
I think you don't want to do that. Any sort of timestamp processing we do will result in garbage.

Can you just convert to 64-bit integer microsecond timestamps and convert back when you get the decoded frames back? I think things will work better like that.

@ghost
Copy link
Author

ghost commented Aug 19, 2015

I think you don't want to do that. Any sort of timestamp processing we do will result in garbage.

Well, it should do no timestamp processing at all. There's no reason for it to do so.

@popcornmix
Copy link
Contributor

If you disable interpolate timestamps and don't feed in frames with OMX_BUFFERFLAG_TIME_IS_DTS set, or tunnel to clocks or use deinterlace components, then yes, you should just get back the timestamps you submitted.

@ghost
Copy link
Author

ghost commented Aug 19, 2015

If I disable interpolation, I'm still getting MMAL_TIME_UNKNOWN. Is that correct?

@popcornmix
Copy link
Contributor

Yes. For whetever reason this file doesn't have a one-to-one mapping of encoded packets to decoded frames. If a second frame is produced from an encoded packet it will have no timestamp associated.

@ghost
Copy link
Author

ghost commented Aug 19, 2015

If a second frame is produced from an encoded packet it will have no timestamp associated.

Well, FFmpeg has no such issues. As I've shown in the first post, MMAL does not output the timestamp of a certain input packet, and outputs a buffer with the wrong timestamp instead (or MMAL_TIME_UNKNOWN if interpolation is disabled). Even more, the data checksums all match between the MMAL decoder and FFmpeg's software decoder, so the output frames are exactly the same, except for the timestamp. (Look at the first command involving showinfo - it has a checksum and plane checksums; both are CRCs of the full image data.)

I can't exclude that the data is broken or that my API usage is wrong, of course, but I'm quite in the dark.

@popcornmix
Copy link
Contributor

Would need our codec expert to explain exactly what the circumstances are where a frame comes back without a timestamp - this frame comes back with the same userdata as a previous frame (cb->used==1) so is marked as timestamp unknown. @deborah-c's not been around for a while but she may be able to comment.

@ghost
Copy link
Author

ghost commented Aug 21, 2015

Here's another sample that shows the issue with higher frequency: https://0x0.st/s0y.mkv

@ghost
Copy link
Author

ghost commented Sep 8, 2015

Well, I still have no idea what could be causing it.

Fact is that it seems to have a problem with some bitstreams as generated by ffmpeg's mp4 to Annex B converter. Both samples have the missing timestamp problem on the packet that is marked as matroska key frame, and whose packet also contains a IDR picture slice NAL (nal_unit_type 5). The libavcodec mp4 to Annex B converter normally inserts the SPS from the extradata before every IDR nal. The sample streams also have a NAL unit type 8 (PPS) in the mkv keyframe packets, so the order is SPS-PPS-IDR. Which should be correct.

Could this be caused by the SPS/PPS being in a different frame as the IDR nal? "Frame" as in MMAL frame boundaries, as delimited by MMAL_BUFFER_HEADER_FLAG_FRAME_START and MMAL_BUFFER_HEADER_FLAG_FRAME_END. I'm trying to set these flags based on the Matroska packet boundaries. One suspicious thing is that the Matroska packets have the bitstream PPS before the keyframe packet with the IDR slice, so both the SPS and PPS end up in the frame before the frame containing the IDR slice. Removing these flags and MMAL_ES_FORMAT_FLAG_FRAMED doesn't seem to help, though. Neither did an attempt to shuffle SPS/PPS into the same frame as the IDR slice.

Does the MMAL decoder have any restrictions here?

Despite the timestamp being sometimes wrong, the decoded image data is always correct.

While we're at it, can the h264 decoder take mp4 style packets instead of requiring Annex B? If so, how?

@deborah-c
Copy link

H.264 defines that given SPS/PPS/Slice, the access unit starts at the first byte of the SPS. Our decoder is quite careful about honoring this, and will associate the timestamp current at that byte (or, actually, a proxy for timestamps, since the core decoder doesn't see actual PTS/DTS values) with the resulting picture's

As a result, the timestamps of the buffer in which the SPS is sent are going to get used, I think. I'm not sure what will happen if one tries to use the frame flags to force a frame boundary at the actual IDR; the flags are essentially treated as extra hints to augment spot-the-NAL-header code, primarily for latency reduction.

The core decoder can indeed accept native 14496-15 data, including the configuration data structure. This is certainly plumbed through video_decode, but I'm not sure how to access it from the ARM side without checking. I'll try to look at this next time I'm in the office. (I'm just back after major illness, so it may take a little while, I'm afraid)

@ghost
Copy link
Author

ghost commented Sep 9, 2015

Thanks for the explanations. I consider this a bug in the ffmpeg mp4-to-Annex B converter.

It would be quite interesting to feed it mp4 style data directly. Looking at the xbmc code, it seems you have to send the extradata in a buffer flagged with MMAL_BUFFER_HEADER_FLAG_CONFIG, but my first attempt at it wasn't overly successful.

@ghost
Copy link
Author

ghost commented Sep 21, 2015

Closing, as it's obviously a FFmpeg bug, and nothing on the MMAL side.

@ghost ghost closed this as completed Sep 21, 2015
@popcornmix
Copy link
Contributor

I believe MMAL auto detects annex B format.
For OpenMAX I had to tell the decoder:
https://github.com/xbmc/xbmc/blob/master/xbmc/cores/omxplayer/OMXVideo.cpp#L616
But I don't believe that is needed with MMAL.

You send the extradata like this: https://github.com/xbmc/xbmc/blob/master/xbmc/cores/dvdplayer/DVDCodecs/Video/MMALCodec.cpp#L499

@ghost
Copy link
Author

ghost commented Sep 22, 2015

OK, so I switched to sending extradata via MMAL_BUFFER_HEADER_FLAG_CONFIG. And... the timestamp problem still happens.

The source data still has the PPS in the "previous" Matroska packet, and I guess the RPI firmware assumes packet == access unit. On the other hand, I don't see why MMAL would have to drop the timestamps for mp4-style data - after all, the packet boundaries are very clearly defined by external means (the container and the MMAL_BUFFER_HEADER_FLAG_FRAME_START/MMAL_BUFFER_HEADER_FLAG_FRAME_END flags).

So is there a chance this wiill be fixed, or should I consider this just an unsolvable firmware issue?

@ghost ghost reopened this Sep 22, 2015
@ghost
Copy link
Author

ghost commented Sep 22, 2015

Also, what's MMAL_ES_FORMAT_T.extradata for? I never could see it having any effect, and kodi doesn't use it either.

@popcornmix
Copy link
Contributor

Are you still using mp4-to-Annex B converter? I feel that is not required and may be introducing a problem.

Setting MMAL_ES_FORMAT_T extradata and extradata_size should be equivalent to using MMAL_BUFFER_HEADER_FLAG_CONFIG, but it only supports limited size (I think MMAL_FORMAT_EXTRADATA_MAX_SIZE=128).

As video codecs sometimes require more than 128 bytes of extradata, it's recommended you use a dedicated packet with MMAL_BUFFER_HEADER_FLAG_CONFIG as that does not have the size limit.

@6by9 may be able to correct me if I'm wrong.

@6by9
Copy link

6by9 commented Sep 22, 2015

Also, what's MMAL_ES_FORMAT_T.extradata for? I never could see it having any effect, and kodi doesn't use it either.

The MMAL header documentation markup actually gives you a worked example that uses extradata.
https://github.com/raspberrypi/userland/blob/master/interface/mmal/mmal.h#L251
It gives you a method of providing the codec header bytes out of band, rather than in buffers. When converted into the IL layer, it is converted into a setParameter(OMX_IndexParamCodecConfig).

Sorry, I'm not a codec expert, and a quick Google has failed me on providing detail of annex B vs MP4 format. A quick read of the code says it supports searching for start codes, or the client guaranteeing one NAL per buffer. I don't know how it handles things beyond that.
(Oddly the encoder supports start codes, or FourByteInterleavedLength, which puts a 32bit length field at the start of every buffer instead of the start code).

@6by9
Copy link

6by9 commented Sep 22, 2015

Comment overlap!

Yes, you're right that it is just another method of providing the header bytes.
If making requests to the GPU, then there is a limit of 128 bytes of extradata due to the IPC.

@6by9
Copy link

6by9 commented Sep 22, 2015

Just checked further, if the MMAL encoding is H264, then the framework looks at the encoding_variant field for either MMAL_ENCODING_VARIANT_H264_DEFAULT or _H264_RAW. These become OMX_NaluFormatStartCodes or OMX_NaluFormatOneNaluPerBuffer respectively. The default case is 0 which is neither of those, however the rest of the code only checks for OneNaluPerBuffer, so it is equivalent to StartCodes.

@ghost
Copy link
Author

ghost commented Sep 22, 2015

Are you still using mp4-to-Annex B converter? I feel that is not required and may be introducing a problem.

No, I removed it. Now I'm feeding one buffer per packet, with the FRAME_START and FRAME_END flags set on each buffer. (If the buffer size is smaller than the packet size, then I split packets into multiple buffers, with the first having FRAME_START set, and the last FRAME_END.)

Setting MMAL_ES_FORMAT_T extradata and extradata_size should be equivalent to using MMAL_BUFFER_HEADER_FLAG_CONFIG, but it only supports limited size (I think MMAL_FORMAT_EXTRADATA_MAX_SIZE=128).

It doesn't seem to work for me, though. If I set the extradata like this, MMAL won't return any decoded frames. (Maybe because it doesn't pick up the extradata and thus isn't able to decode anything.) I used this code:

    if (avctx->extradata_size) {
        if ((status = mmal_format_extradata_alloc(format_in, avctx->extradata_size)))
            goto fail;
        format_in->extradata_size = avctx->extradata_size;
        memcpy(format_in->extradata, avctx->extradata, format_in->extradata_size);
    }

    if ((status = mmal_port_format_commit(decoder->input[0])))
        goto fail;

(The normal format setup, such as setting the video codec, is above this code.)

The extradata is only 42 bytes with the file I tested. If I use MMAL_BUFFER_HEADER_FLAG_CONFIG, it works as expected. (Except the timestamp issue still happens.)

The MMAL header documentation markup actually gives you a worked example that uses extradata.
https://github.com/raspberrypi/userland/blob/master/interface/mmal/mmal.h#L251

Whoops, sorry, I've never seen this. I always attempted to look at the definitions directly. This still doesn't answer any of my questions, though.

Sorry, I'm not a codec expert, and a quick Google has failed me on providing detail of annex B vs MP4 format. A quick read of the code says it supports searching for start codes, or the client guaranteeing one NAL per buffer. I don't know how it handles things beyond that.

They are slightly different - mp4 prefixes NALs with length fields (and stores SPS etc. separately), while annex B has this start code business. (Possibly for easier resyncing in broken streams?) I don't know too much about it either.

But now it seems MMAL fundamentally cares about how additional NALs like PPS are distributed over "access units" - so even if the mp4-style packet format is used, it doesn't really care about packet boundaries, and thus breaks the association between timestamps and packets. It's quite possible that these files violate the mp4 standard (well, they're not even mp4, but Matroska), and that MMAL is not actually doing anything wrong strictly speaking. It's still somewhat annoying though, because nothing else seems to have issues with these files.

@JamesH65
Copy link
Contributor

@6by9 reading through this issue, its 2 years since last comment, have we made any timestamp changes in that time? Are we likely to make any changes to this area?

@6by9
Copy link

6by9 commented Dec 19, 2017

Not that I'm aware of.

@JamesH65
Copy link
Contributor

Closing due to lack of activity. Reopen if you feel this issue is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants