Frame reordering sometimes leads to dropped frames

[mkver]

Hello,

as you know, coding order and presentation order do not coincide if one uses b-frames in which case the frames need to be reordered by the decoder which therefore has to know (or guess) how high the reorder count is (the reorder count (of frame X) is the number of frames preceding X in coding order, but following it in display order) in order to know how many frames to buffer. ffmpeg seems to use the following strategy for the reorder count of H.264 streams (or actually for H.264 in Matroska -- I haven't checked other containers which often convey dts explicitly whereby Matroska only conveys decoding order and PTS): If the stream indicates its reorder count (in the VUI (video usability information) data, an optional part of the Sequence Parameter Set (SPS)), that number is used; if not, the first GOP is analyzed and its reorder count is used. If this initial estimate is too low, then ffmpeg drops some frames and increases its estimate of the reorder count (and emits a warning "Increasing reorder buffer to 2" [in my case 2, others may of course differ] if you are using ffmpeg directly). (Thanks to nevcairiel who explained this to me here.)
Here is a file where this guessing logic fails. Here is the beginning of mkvinfo's output for it:

I frame, track 1, timecode 0 (00:00:00.000), size 85243, adler 0x2f40d0ce
P frame, track 1, timecode 120 (00:00:00.120), size 47970, adler 0x6f500910
B frame, track 1, timecode 20 (00:00:00.020), size 303, adler 0x8d8b8f45
P frame, track 1, timecode 40 (00:00:00.040), size 11450, adler 0xa43210b8
P frame, track 1, timecode 60 (00:00:00.060), size 11158, adler 0xe954abf0
P frame, track 1, timecode 80 (00:00:00.080), size 12031, adler 0x82d06695
P frame, track 1, timecode 100 (00:00:00.100), size 11849, adler 0x831b333b
P frame, track 1, timecode 240 (00:00:00.240), size 50205, adler 0xec238176
B frame, track 1, timecode 140 (00:00:00.140), size 492, adler 0x6606f795
P frame, track 1, timecode 160 (00:00:00.160), size 11334, adler 0xa37751ce
P frame, track 1, timecode 180 (00:00:00.180), size 11128, adler 0x73c4c2b6
P frame, track 1, timecode 200 (00:00:00.200), size 10763, adler 0xbaaae837
P frame, track 1, timecode 220 (00:00:00.220), size 10683, adler 0xfe01ef8c
P frame, track 1, timecode 360 (00:00:00.360), size 51920, adler 0x63124640
B frame, track 1, timecode 260 (00:00:00.260), size 540, adler 0xab3109ab
P frame, track 1, timecode 280 (00:00:00.280), size 11247, adler 0xfd3801bb
P frame, track 1, timecode 300 (00:00:00.300), size 11001, adler 0x1e2f7334
P frame, track 1, timecode 320 (00:00:00.320), size 11549, adler 0x999e9a0c
P frame, track 1, timecode 340 (00:00:00.340), size 11377, adler 0x235a08ae
P frame, track 1, timecode 480 (00:00:00.480), size 51066, adler 0x23e15f29
B frame, track 1, timecode 380 (00:00:00.380), size 804, adler 0x08649cb8
P frame, track 1, timecode 400 (00:00:00.400), size 10488, adler 0x2f5652c3
P frame, track 1, timecode 420 (00:00:00.420), size 10327, adler 0x52621163
P frame, track 1, timecode 440 (00:00:00.440), size 10710, adler 0x4b9e0b88
P frame, track 1, timecode 460 (00:00:00.460), size 10552, adler 0x3bd7ae49
I frame, track 1, timecode 640 (00:00:00.640), size 70447, adler 0x839e176a
B frame, track 1, timecode 560 (00:00:00.560), size 19762, adler 0xecda96f0
B frame, track 1, timecode 500 (00:00:00.500), size 2457, adler 0xff55ca57
P frame, track 1, timecode 520 (00:00:00.520), size 8209, adler 0x5f12031d
P frame, track 1, timecode 540 (00:00:00.540), size 8013, adler 0x651b815f
P frame, track 1, timecode 580 (00:00:00.580), size 783, adler 0xd8ae8e63
P frame, track 1, timecode 600 (00:00:00.600), size 7181, adler 0xfa6a1093
P frame, track 1, timecode 620 (00:00:00.620), size 6995, adler 0x93faa2b9

As one can see, the reorder count is always one or zero in the first GOP. But this changes in the second GOP: The frames 25-28 in (0-based) display order with timestamps 500, 520 and 540 have a reorder count of two so ffmpeg has underestimated the reorder count and drops the frames 25-27; so does ffms2. (I somewhere read that by adding -strict 1 changes ffmpeg's guessing logic and uses a very high value for the reorder count so that one doesn't have dropped frames. It seems to be true.) Contrary to ffmpeg (which emits one frame at 480 and another at 560) ffms2 repeats the frame with timestamp 560 four times to make up for the dropped frames. Here is a sample.
In order to show that this is really about frame reordering, I extracted the H.264 stream from the Matroska file and changed the SPS so that it now includes the entry for reorder frames. (I used three reorder frames, although two would have been enough.) Afterwards I remuxed to Matroska. Here is the result. Now ffmpeg and ffmpeg based players played the file flawlessly and also ffms2 did not drop frames. Here is the proof.
I see two potential solutions for this problem: First, you could use a very high number of potential frame reorderings like the --strict 1 switch. But there is also a second solution: One could get the necessary value for reorder count as a byproduct of the indexing which means that there is no need to guess anything.
Thanks for all your efforts so far and also for the time you invest into this bug report.

Greetings
mkver

[myrsloik]

Interesting observation and useful sample, it will be investigated some day.

[myrsloik]

Applied the fix and now sets the reorder size to max always to avoid it.