PGS subtitles in MKV container and seeking

[mkver]

Hello nevcairiel,

since release 0.65 you support the matroska-cues for subtitles. And for srt and vobsubs this works as expected. But not for HDMV PGS-subtitles. They are usually muxed without a duration element (often in SimpleBlocks) and are ought to be shown until the next packet is encountered in the file. Therefore e.g. MKVToolNix doesn't write the CueDuration element for them at all (and during my testing FFMPEG's muxer wrote a garbage duration of zero in the file). And it doesn't have to, because it's irrelevant; after all, the seeking strategy for these subtitles should be "find the subtitle cues with the highest timecode lower or equal to the time we are seeking to, seek to the (simple)block containing the subtitles and display them until another subtitle packet is encountered" and this works without CueDuration. MPC-HC with LAV Splitter (and decoder, but that should be irrelevant) doesn't show these subtitles upon seeking when it should. Can you fix this? Or is this a problem upstream (in FFMPEG)?
Furthermore there is a problem with the other permitted way to mux PGS subs (namely muxing with durations), too: The durations are ignored in this case and the subtitles are shown until the next packet is encountered (or until a seek). At least the subtitles are properly shown upon seeking (if the cues contain the right CueDuration element).* I actually don't know if this is an issue in LAV filter or in the subtitle renderer.
Here is a sample file.

Thanks in advance
Andi

*: The possibility of this second way of muxing actually implies that the seeking strategy I mentioned above needs to be modified to include a check whether the found subtitle packet has a duration to determine whether the subtitles should really be displayed.

[Nevcairiel]

The entire point of the Subtitle Cues is that you can find out which subtitles to display without requiring to demux large parts of the files, in fact you should be able to judge this from the Cue elements alone and nothing else - the new elements have been designed just for that.

Hence, if this feature is wanted to be used, the CueDuration elements should be written, even if subtitle decoders don't typically use the duration values otherwise (Cue durations are unrelated to the block durations anyway).

Codec-specific logic is not specified or wanted whatsoever.

For the record, both the .mks files in the sample you linked seem to contain CueDuration elements, which seem to match the timestamp difference between the subtitle blocks, so it would seem they are present and written properly.

[mkver]

1. The way current muxers work one can find out which subtitles to display without demuxing large parts of the file, but one needs to codec-specific logic for that, namely the seeking strategy I outlined in the first post.
2. Given that you rule out codec-specific logic you say that the muxers should be changed. Is that right?
3. The file "St.Vin.BlockDuration.mks" does contain CueDuration elements and BlockDuration elements (which match), but they don't match the timestamp difference between the subtitles blocks, because the small blocks which signal to end the display of subtitles have been removed (after all, the time is already contained in the BlockDuration).

[Nevcairiel]

If muxers don't write CueDuration or write broken values, then those should be changed. yes. Like I said however, both your mks files do contain CueDuration, so none of your samples is actually broken. So at least mkvtoolnix seems to write these elements for PGS then.

[mkver]

No, MKVToolNix usually doesn't write these elements. It writes CueRelativePosition elements by default, but it only writes the CueDuration elements if the subtitle packets of the input file contained BlockDuration elements; and regardless of whether the PGS-subtitles in an input file contain BlockDuration elements or not, they are discarded upon remuxing (i.e. if you remux "St.Vin.BlockDuration.mks", the only place where the durations will be stored are in the cues (which of course means that the valid duration for these packets is the difference between timestamps of the next and the current packet, i.e. the duration is lost)). There is a way to preserve the BlockDurations (with the --engage force_passthrough_packetizer switch), but that's not the default behaviour (and it has sideeffects: it disables lacing for all tracks).
Both files contain CueDurations because of the way they were created: "St.Vin.Original.mks" has been created by remuxing with mkvmerge a file written by MakeMKV (which writes BlockDurations for PGS-subtitles and does not discard the packets intended to stop playback of the previous subtitle). Therefore it contains CueDurations, but not BlockDurations. This file has then been modified (with a hex editor) in such a way that the duration is indicated by BlockDuration elements and not by the stop packets. This file has then been remuxed via MKVToolNix and the --engage force_passthrough_packetizer switch. (My original intention was to test MKVToolNix and video players). If you mux St.Vin.Original.mks with MKVToolNix to any video file, the subtitles are not shown upon seeking (but during normal playback the subtitles are properly shown). Given MKVToolNix popularity, this is the usual case (i.e. it encompasses probably a majority of matroska-files with pgs-subtitles in existence). If you mux St.Vin.BlockDuration.mks to any video file with the "--engage force_passthrough_packetizer" the subtitles are properly shown upon seeking, but the still existing durations are ignored (i.e. the subtitles are shown until the next subtitle packet is encountered). Given that to the best of my knowledge no muxer strips away the stop packets, this file is highly unusual (as said before: it's designed as a test file).

[Nevcairiel]

Specially crafted test files that behave entirely differently to real-world files are not particularly useful when you try to report issues with real-world files, just saying.

In any case, I think the point is clear. Why should every demuxer need to scan through the subtitles to determine their duration when a muxer could do that when its writing them anyway, and write the appropriate CueDuration element that the specificiation contains for this exact purpose?

[mkver]

Actually I thought I to be useful to test unusual, but standards-compliant edge cases to see if muxers/players honor the specs or support only the usual way of doing things.
mkvmerge's reason for its way of doing things is probably that the CueDuration elements just use space and aren't needed for the usual way (after all the PGS-subtitles written by mkvmerge usually don't contain durations and therefore one does not need to scan through the subtitles, one just needs to scan the cues and seek to the packet with the highest timecode lower or equal than the point one is seeking to; I don't see a reason why this approach should lead to more seeks than the approach with CueDuration).