New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MMAL video decoder sometimes outputs corrupted timestamps #456
Comments
What does |
Looking at the file with kodi using mmal:
and I get back:
So it looks okay here. There is something different about 10.427. It is an IDR frame and it comes back from the codec with timestamp unknown. It's a bit beyond my knowledge to say why, but this normally occurs when there are multiple frames in a packet (e.g. the vfw/avi packed bitstream hack). However MMAL on Kodi sets MMAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS so the unknown timestamp gets set to the correct value. Without that I'd expect you to see MMAL_TIME_UNKNOWN as the pts value. This assumes you are using latest rpi-update firmware. There were timestamp related bugs in earlier versions. |
I haven't updated the firmware quite in a while, so maybe that's the problem and it's fixed now. Currently I can't access my RPI though. (Both LEDs are lit without blinking, internet says it's a SD card problem, but both cards I tried work with the PC card reader.) |
Do you see rainbow splash screen on boot? |
No, the screen simply remained off all the time. Somehow I got it to start again (tried various combinations of disconnecting power and reinserting the SD card - maybe some physical issue). The firmware was old, but after updating, I got the same behavior. The firmware version is now:
I've also tried: mmal_port_parameter_set_boolean(decoder->output[0], MAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS, 1) But it didn't seem to help. Note that the code does handle |
MMAL_PARAMETER_VIDEO_INTERPOLATE_TIMESTAMPS does default to enabled, so you'd have to set it to 0 to disable. Are you setting the framerate of the MMAL decoder? Having a bad value in there may cause unwanted effects with timestamp interpolation. |
It's set to a dummy value. |
What value? Does it behave better if not set or set to correct value? |
It's 24000/1001. I can test changing it tomorrow; I'll also test whether disabling timestamp interpolation returns I've been wondering what the framerate field is about, since a decoder doesn't need it in theory. I'm also not sure what MMAL would do with the timestamps - since MMAL doesn't know the unit (and libavcodec also doesn't), they have any real meaning and are opaque. |
There are a few cases when timestamps are used by video_decode: DTS timestamps are a problem. If your container has DTS but not PTS timestamps and includes B frames, then passing them through decoder will reorder the timestamps in a way you probably can't make use of (without parsing the video stream sufficiently to determine what reordering has occurred). If you set the OMX_BUFFERFLAG_TIME_IS_DTS flag, then videodecode will try to associate unreordered timestamps with the frames that come back. However this is only possible in certain conditions (e.g. number of encoded frames matching number of decoded frames). |
Changing the framerate passed to MMAL while interpolation is enabled doesn't change the bogus timestamp that is output.. If I disable timestamp interpolation, the MMAL decoder actually returns |
If you can provide an example app that produces an unexpected interpolated timestamp for this frame I could debug it. (Can just be a binary - I don't need to debug the arm side). |
Would a statically built mpv binary do? |
(By which I mean it would only link to a few libs installed on Raspbian by default.) |
Anyway, here's a mpv binary compiled under Raspbian: https://0x0.st/s6U.exe (the exe suffix was forced by the site) Rename to mpv, chmod, start playback with |
The timestamps I see are suspicious:
I assume this is what you mean by "IEEE doubles reinterpret-casted to int64". Can you just convert to 64-bit integer microsecond timestamps and convert back when you get the decoded frames back? I think things will work better like that. |
Well, it should do no timestamp processing at all. There's no reason for it to do so. |
If you disable interpolate timestamps and don't feed in frames with OMX_BUFFERFLAG_TIME_IS_DTS set, or tunnel to clocks or use deinterlace components, then yes, you should just get back the timestamps you submitted. |
If I disable interpolation, I'm still getting |
Yes. For whetever reason this file doesn't have a one-to-one mapping of encoded packets to decoded frames. If a second frame is produced from an encoded packet it will have no timestamp associated. |
Well, FFmpeg has no such issues. As I've shown in the first post, MMAL does not output the timestamp of a certain input packet, and outputs a buffer with the wrong timestamp instead (or I can't exclude that the data is broken or that my API usage is wrong, of course, but I'm quite in the dark. |
Would need our codec expert to explain exactly what the circumstances are where a frame comes back without a timestamp - this frame comes back with the same userdata as a previous frame (cb->used==1) so is marked as timestamp unknown. @deborah-c's not been around for a while but she may be able to comment. |
Here's another sample that shows the issue with higher frequency: https://0x0.st/s0y.mkv |
Well, I still have no idea what could be causing it. Fact is that it seems to have a problem with some bitstreams as generated by ffmpeg's mp4 to Annex B converter. Both samples have the missing timestamp problem on the packet that is marked as matroska key frame, and whose packet also contains a IDR picture slice NAL (nal_unit_type 5). The libavcodec mp4 to Annex B converter normally inserts the SPS from the extradata before every IDR nal. The sample streams also have a NAL unit type 8 (PPS) in the mkv keyframe packets, so the order is SPS-PPS-IDR. Which should be correct. Could this be caused by the SPS/PPS being in a different frame as the IDR nal? "Frame" as in MMAL frame boundaries, as delimited by MMAL_BUFFER_HEADER_FLAG_FRAME_START and MMAL_BUFFER_HEADER_FLAG_FRAME_END. I'm trying to set these flags based on the Matroska packet boundaries. One suspicious thing is that the Matroska packets have the bitstream PPS before the keyframe packet with the IDR slice, so both the SPS and PPS end up in the frame before the frame containing the IDR slice. Removing these flags and MMAL_ES_FORMAT_FLAG_FRAMED doesn't seem to help, though. Neither did an attempt to shuffle SPS/PPS into the same frame as the IDR slice. Does the MMAL decoder have any restrictions here? Despite the timestamp being sometimes wrong, the decoded image data is always correct. While we're at it, can the h264 decoder take mp4 style packets instead of requiring Annex B? If so, how? |
H.264 defines that given SPS/PPS/Slice, the access unit starts at the first byte of the SPS. Our decoder is quite careful about honoring this, and will associate the timestamp current at that byte (or, actually, a proxy for timestamps, since the core decoder doesn't see actual PTS/DTS values) with the resulting picture's As a result, the timestamps of the buffer in which the SPS is sent are going to get used, I think. I'm not sure what will happen if one tries to use the frame flags to force a frame boundary at the actual IDR; the flags are essentially treated as extra hints to augment spot-the-NAL-header code, primarily for latency reduction. The core decoder can indeed accept native 14496-15 data, including the configuration data structure. This is certainly plumbed through video_decode, but I'm not sure how to access it from the ARM side without checking. I'll try to look at this next time I'm in the office. (I'm just back after major illness, so it may take a little while, I'm afraid) |
Thanks for the explanations. I consider this a bug in the ffmpeg mp4-to-Annex B converter. It would be quite interesting to feed it mp4 style data directly. Looking at the xbmc code, it seems you have to send the extradata in a buffer flagged with MMAL_BUFFER_HEADER_FLAG_CONFIG, but my first attempt at it wasn't overly successful. |
Closing, as it's obviously a FFmpeg bug, and nothing on the MMAL side. |
I believe MMAL auto detects annex B format. You send the extradata like this: https://github.com/xbmc/xbmc/blob/master/xbmc/cores/dvdplayer/DVDCodecs/Video/MMALCodec.cpp#L499 |
OK, so I switched to sending extradata via MMAL_BUFFER_HEADER_FLAG_CONFIG. And... the timestamp problem still happens. The source data still has the PPS in the "previous" Matroska packet, and I guess the RPI firmware assumes packet == access unit. On the other hand, I don't see why MMAL would have to drop the timestamps for mp4-style data - after all, the packet boundaries are very clearly defined by external means (the container and the MMAL_BUFFER_HEADER_FLAG_FRAME_START/MMAL_BUFFER_HEADER_FLAG_FRAME_END flags). So is there a chance this wiill be fixed, or should I consider this just an unsolvable firmware issue? |
Also, what's MMAL_ES_FORMAT_T.extradata for? I never could see it having any effect, and kodi doesn't use it either. |
Are you still using mp4-to-Annex B converter? I feel that is not required and may be introducing a problem. Setting MMAL_ES_FORMAT_T extradata and extradata_size should be equivalent to using MMAL_BUFFER_HEADER_FLAG_CONFIG, but it only supports limited size (I think MMAL_FORMAT_EXTRADATA_MAX_SIZE=128). As video codecs sometimes require more than 128 bytes of extradata, it's recommended you use a dedicated packet with MMAL_BUFFER_HEADER_FLAG_CONFIG as that does not have the size limit. @6by9 may be able to correct me if I'm wrong. |
The MMAL header documentation markup actually gives you a worked example that uses extradata. Sorry, I'm not a codec expert, and a quick Google has failed me on providing detail of annex B vs MP4 format. A quick read of the code says it supports searching for start codes, or the client guaranteeing one NAL per buffer. I don't know how it handles things beyond that. |
Comment overlap! Yes, you're right that it is just another method of providing the header bytes. |
Just checked further, if the MMAL encoding is H264, then the framework looks at the encoding_variant field for either MMAL_ENCODING_VARIANT_H264_DEFAULT or _H264_RAW. These become OMX_NaluFormatStartCodes or OMX_NaluFormatOneNaluPerBuffer respectively. The default case is 0 which is neither of those, however the rest of the code only checks for OneNaluPerBuffer, so it is equivalent to StartCodes. |
No, I removed it. Now I'm feeding one buffer per packet, with the FRAME_START and FRAME_END flags set on each buffer. (If the buffer size is smaller than the packet size, then I split packets into multiple buffers, with the first having FRAME_START set, and the last FRAME_END.)
It doesn't seem to work for me, though. If I set the extradata like this, MMAL won't return any decoded frames. (Maybe because it doesn't pick up the extradata and thus isn't able to decode anything.) I used this code: if (avctx->extradata_size) {
if ((status = mmal_format_extradata_alloc(format_in, avctx->extradata_size)))
goto fail;
format_in->extradata_size = avctx->extradata_size;
memcpy(format_in->extradata, avctx->extradata, format_in->extradata_size);
}
if ((status = mmal_port_format_commit(decoder->input[0])))
goto fail; (The normal format setup, such as setting the video codec, is above this code.) The extradata is only 42 bytes with the file I tested. If I use MMAL_BUFFER_HEADER_FLAG_CONFIG, it works as expected. (Except the timestamp issue still happens.)
Whoops, sorry, I've never seen this. I always attempted to look at the definitions directly. This still doesn't answer any of my questions, though.
They are slightly different - mp4 prefixes NALs with length fields (and stores SPS etc. separately), while annex B has this start code business. (Possibly for easier resyncing in broken streams?) I don't know too much about it either. But now it seems MMAL fundamentally cares about how additional NALs like PPS are distributed over "access units" - so even if the mp4-style packet format is used, it doesn't really care about packet boundaries, and thus breaks the association between timestamps and packets. It's quite possible that these files violate the mp4 standard (well, they're not even mp4, but Matroska), and that MMAL is not actually doing anything wrong strictly speaking. It's still somewhat annoying though, because nothing else seems to have issues with these files. |
@6by9 reading through this issue, its 2 years since last comment, have we made any timestamp changes in that time? Are we likely to make any changes to this area? |
Not that I'm aware of. |
Closing due to lack of activity. Reopen if you feel this issue is still relevant. |
I'm not really sure what's happening here. But it almost looks like MMAL is outputting corrupted timestamps every 10 seconds with a specific sample I have.
Here are the first 15 seconds or so of the sample: https://0x0.st/sV_.mkv
With
ffmpeg -c:v h264_mmal -i sample.mkv -vf showinfo -f null -
I get this at around the 10 second mark (use FFmpeg git master as of now; older versions have a bug):Frame number 250 outputs an odd timestamp. Frame number 251 is back to normal.
It happens after the next 250 frames too:
(not included in the sample, but I can upload the full one if needed)
But the packet timestamp seems to be fine:
(from
ffprobe sample.mkv -show_packets
)The FFmpeg h264 software decoder has no issues here, it outputs the frame like this:
(from
ffprobe -show_frames -select_streams 1 sample.mkv
)Is it possible that MMAL is doing something wrong here?
(Side note: ffprobe is a useful tool, but you can't force the video decoder yet - so it can't be used with the mmal wrapper.)
The text was updated successfully, but these errors were encountered: