Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance

[davemevans]

In order to support use cases such as trick play, as well as very low latency playback, it may sometimes be useful to be able to allow the media time to keep incrementing at all costs whether or not there is any media to be played.

Ways in which I can imagine this working could be:
a) signal a SourceBuffer as potentially containing sparse data and not to stall the timeline during unbuffered ranges (exact behaviour to be defined eg hold last frame)
b) explicitly 'append' an 'empty' range

I appreciate that this may be better solved by the media element, or at application level, but I think it is at least worth considering in MSE.

I intend this is to be considered a VNext use case.

[mwatson2]

@bbcrddave I understand there is a use-case for this for live services, where the desired behaviour when the buffer runs empty is not to fall behind "real-time" but to pause and then jump forward once new data is available.

But could you elaborate on the trick play and low latency playback use-cases ?

[jyavenard]

sounds similar to #21
(and #133)

[davemevans]

The 'low-latency' use case is essentially the problem you describe where data may not be available in time for whatever reason and staying at the live edge is deemed more critical than having moving pictures. My choice of words was poor because this isn't about decode latency (so not the same as #21 etc) but staying at the live edge at all times.

For trick play, it may be difficult to download data required to maintain the buffer as playback speed increases. It may be useful to append only some segments (every xth for example), despite the potential UX problems, or even partial segments where possible, and leave gaps in the buffer which are played through.

[wolenetz]

Thank you for filing this issue. While partially a duplicate of #133 and #21, I'll keep this one independent in VNext to support the specific use case mentioned here.

[jdsmith3000]

I think we generally agree there may be a need for low latency. I can also imagine that trick play might benefit from skipping segments during appends, but as you note, the app could probably do this perhaps just as well. Can you provide evidence of issues here that are difficult to solve with the current MSE API?

[chanceym]

Using MSE to broadcast live video streams has great potential but lacks some key features that must be addressed to really be a viable solution. First the ability to skip over missing frames (play through gaps) is a key feature that would need to be addressed. Second the ability to append sourcebuffers with video data received from one of many video sources (passing the permission from one participant to another to transmit video), currently I have to tare-down a sourcebuffer and rebuild it every time the video is passed, causing a flash (no desirable). I tried using 'Sequence' mode to solve these issues, but got unexpected video drift that was worse than what I tried to solve.

[michaeljweaver]

@wolenetz do you know if this feature request will be included in the work being done for the referenced #232 ?
That's tracked by https://bugs.chromium.org/p/chromium/issues/detail?id=963717

[wolenetz]

#232 is for modes. This (#160) is to let app see precisely what each SourceBuffer thinks is a gap (perhaps also expanding on that with the ability to inspect buffered ranges at the track level, for instance, for multi-track SourceBuffers), and to let apps specify:

1. How much tolerance the MSE-extended HTMLMediaElement.buffered will have to be able to coalesce any SourceBuffer gaps (e.g. "play through any gap that is smaller than 0.75 seconds") and not stall,
2. and/or what HTMLMediaElement playback will do when it encounters a gap (after (1)'s coalescence, if any) with a buffered range later (such as play silence for missing audio, frozen last frame for missing video while letting clock advance at playbackRate? Or seek to the next point where all active tracks have buffered media? Or seek to the earliest next point where an active track has buffered media (e.g. for commonly jagged A/V segment buffered range starts) and play silence/last frame for any other track until it also gets buffered.

This has been discussed multiple times (FOMS 2018, FOMS NYC March 2019, various F2F including recent TPAC Sep 2019 media WG F2F), and will likely become a vNext feature to incubate soon.

[wolenetz]

Per whatwg/html#6359, we'll likely need to help UAs and apps understand and customize audio vs video-specific playback behavior on network underflow (buffered range gap in each track, here) as part of this solution.

[chrisn]

Briefly discussed during 8 Nov 2022 Media WG meeting: https://www.w3.org/2022/11/08-mediawg-minutes.html#t05

[wolenetz]

This came up again at FOMS2023. It's absolutely a valid issue that still needs to be solved.
Meanwhile, there is an ugly hack that might help coalesce unintended gaps in Chromium SourceBuffers: the coalescence heuristic currently used there is based on the largest frame duration buffered so far in that particular track in that particular SourceBuffer. Auto-coalescence of gaps can thus be achieved by manipulation of the inputs to this heuristic:

1. For each track, first append a simple keyframe with huge duration. Ensure it is buffered. Then remove it from the SourceBuffer (or overlap it in your first appends to that track).
2. This should trick the heuristic.

Yes, this is ugly: non-interoperable. Not guaranteed to continue working. Adds overhead to time-to-first-frame (async buffering and removal).
But since this issue has not yet been fixed, and it is hurting MSE users, this hack might help at least on Chromium based browsers.

[dsilhavy]

This came up again at FOMS2023. It's absolutely a valid issue that still needs to be solved. Meanwhile, there is an ugly hack that might help coalesce unintended gaps in Chromium SourceBuffers: the coalescence heuristic currently used there is based on the largest frame duration buffered so far in that particular track in that particular SourceBuffer. Auto-coalescence of gaps can thus be achieved by manipulation of the inputs to this heuristic:

1. For each track, first append a simple keyframe with huge duration. Ensure it is buffered. Then remove it from the SourceBuffer (or overlap it in your first appends to that track).
2. This should trick the heuristic.

Yes, this is ugly: non-interoperable. Not guaranteed to continue working. Adds overhead to time-to-first-frame (async buffering and removal). But since this issue has not yet been fixed, and it is hurting MSE users, this hack might help at least on Chromium based browsers.

Thanks for this, really interesting. I would like to share my observations:

Test setup

I created a media segment with a single IDR frame and a duration of 4 seconds. Note that earliest_presentation_time= 720000 so I am setting sourceBuffer.timestampOffset = -8; to start from 0 in the SourceBuffer:

Problem encountered

Now simply appending the corresponding init segment and the above media segment to the SourceBuffer leads to a buffered range from 0 - 4 as expected. But when removing the buffered range and hitting "play" the current time of the video element is still progressing although there is no valid buffered range:

As a comparison, Firefox is behaving as expected:

I also did not manage to replace the buffered IDR frame with new content. The new content only becomes visible after the segment with the IDR frame has finished (after 4 seconds)

Solution

It looks like a simple seek (e.g. seeking to 0.1) solves the previously described problem:

As @wolenetz described Chrome is playing over gaps in my tests afterwards. In this example, there was a gap between 2-4 seconds:

For comparison Firefox stalls after 2 seconds:

[dsilhavy]

I put my findings in a blog post: https://websites.fraunhofer.de/video-dev/being-trapped-in-a-gap-with-big-buck-bunny/ : Happy to get feedback, maybe I also missed something in my implementation and the gap workaround can be used without a seek.

[wolenetz]

I put my findings in a blog post: https://websites.fraunhofer.de/video-dev/being-trapped-in-a-gap-with-big-buck-bunny/ : Happy to get feedback, maybe I also missed something in my implementation and the gap workaround can be used without a seek.

@dsilhavy Nice write-up! It looks like the "ugly hack" is working for you for video. Seek is understandably a "fix" for the Chromium media pipeline having started prerolling the decoder with the (later removed) media. In this case, the prerolling is aborted by a seek to a time that is not currentTime. Without the seek / preroll abort, the Chrome media pipeline will still accept the newly buffered content into the timeline, but the renderer will throw away decode output with earlier presentation time than what was expected to follow the prerolled content. At least, I think this is what is happening.

To avoid needing to seek, try buffering the fake content elsewhere in the timeline (say, far in the future), then remove it, optionally adjust the MediaSource duration, and then append the real content. How far in the future? This is another point of lack of interop precision. I'd suggest more than thrice the manufactured frame's duration, and no less than 10 seconds, to hopefully avoid prerolling any decode of the fake content.

While considering the "ugly hack" further, I recalled more ugly details. The hack will likely lead to issues even now in Chromium when attempting to auto-coalesce gaps in an audio track's buffered ranges for manifold reasons:

1. Not all Chromium bytestream format parsers obey a fake audio frame's duration: For example, with unencrypted opus in webm, the parser attempts to resolve lack of duration coded frame precision inherent in the webm container itself by violating layers and parsing the opus bitstream. This may complicate or prevent ability to buffer an artificially huge-duration audio coded frame in Chromium. Further workaround in such case might be to parse an artificially large-duration audio coded frame from, say, an mp4 track, and then use changeType() to switch to webm parsing before buffering the real content. EME+changeType() can have complications preventing this. And it would be best to not let the media pipeline preroll the fake audio's decode (so append it far in the future as described earlier in this post) unless its sample rate (and maybe also channel configuration) matches the real content (since, IIRC, the preroll configures the audio sink in such a way that downsampling of the real content later might happen if the fake content had a lower sample rate).
2. Auto-coalesced, but still real in terms of the actual aggregate duration of decoded samples in the range being less than what the range claims, gaps in the audio timeline could lead to scenarios where currentTime doesn't agree with the presentation time of the actual decoded content rendered at that time, and/or where loss of A/V sync occurs.

There are likely other caveats, further motivating getting this spec issue fixed and implemented.

A couple other notes:

1. The sidx box contents are ignored by MSE. Rather, the dts and cts of samples are determined from traf and trun boxes, if I recall correctly. I assume those fragment boxes in Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance #160 (comment) were consistent with the sidx such that the IDR frame was buffered at presentation time range [0,4) after being adjusted to begin at time 0 using the timestampOffset of -8 before it was appended.
2. Understandably, your blog post noted me as being at Google (as the MSE spec working draft still did until we fixed that very recently in Update current wolenetz editor data #333): I'm an ex-Googler at the moment, though still am an MSE spec co-editor as a W3C Invited Expert to the Media WG.