-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance #160
Comments
@bbcrddave I understand there is a use-case for this for live services, where the desired behaviour when the buffer runs empty is not to fall behind "real-time" but to pause and then jump forward once new data is available. But could you elaborate on the trick play and low latency playback use-cases ? |
The 'low-latency' use case is essentially the problem you describe where data may not be available in time for whatever reason and staying at the live edge is deemed more critical than having moving pictures. My choice of words was poor because this isn't about decode latency (so not the same as #21 etc) but staying at the live edge at all times. For trick play, it may be difficult to download data required to maintain the buffer as playback speed increases. It may be useful to append only some segments (every xth for example), despite the potential UX problems, or even partial segments where possible, and leave gaps in the buffer which are played through. |
I think we generally agree there may be a need for low latency. I can also imagine that trick play might benefit from skipping segments during appends, but as you note, the app could probably do this perhaps just as well. Can you provide evidence of issues here that are difficult to solve with the current MSE API? |
Using MSE to broadcast live video streams has great potential but lacks some key features that must be addressed to really be a viable solution. First the ability to skip over missing frames (play through gaps) is a key feature that would need to be addressed. Second the ability to append sourcebuffers with video data received from one of many video sources (passing the permission from one participant to another to transmit video), currently I have to tare-down a sourcebuffer and rebuild it every time the video is passed, causing a flash (no desirable). I tried using 'Sequence' mode to solve these issues, but got unexpected video drift that was worse than what I tried to solve. |
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473}
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473}
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473}
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991
This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Reviewed-by: Dan Sanders <sandersd@chromium.org> Cr-Commit-Position: refs/heads/master@{#643473}
@wolenetz do you know if this feature request will be included in the work being done for the referenced #232 ? |
#232 is for modes. This (#160) is to let app see precisely what each SourceBuffer thinks is a gap (perhaps also expanding on that with the ability to inspect buffered ranges at the track level, for instance, for multi-track SourceBuffers), and to let apps specify:
This has been discussed multiple times (FOMS 2018, FOMS NYC March 2019, various F2F including recent TPAC Sep 2019 media WG F2F), and will likely become a vNext feature to incubate soon. |
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: fc846bf83bfdae75b97fec2fb139b4b88dc9be7f
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: c679de39ad4dbaafeee76985263d5c11c07de772
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: fc846bf83bfdae75b97fec2fb139b4b88dc9be7f
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: c679de39ad4dbaafeee76985263d5c11c07de772
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: fc846bf83bfdae75b97fec2fb139b4b88dc9be7f
…work with MseBufferByPts, a=testonly Automatic update from web-platform-tests MSE: Fix mediasource-changetype-play to work with MseBufferByPts This web-platform-test exercises changeType as it splice-overlaps pairs of audio or video media streams at varying offsets in the presentation timeline. Splice-overlapping an out-of-order decode stream (such as the test AVC MP4 media) at arbitrary times can, per spec, drop significant decode dependencies from a partially-overlapped GOP such that a buffered range gap could result. This change is more careful about where it performs splice-overlaps when the overlapped media is out-of-order-decode, adjusting the splice point to be at or very near to the next overlapped keyframe. This prevents removing out-of-order non-keyframes and their dependents from the overlapped media such that no buffered range gap nor playback stall should result. Note that Chromium is sensitive to such out-of-order buffering overlaps with the new, compliant, MseBufferByPts behavior. Fixing w3c/media-source#160 could greatly simplify this problem by allowing apps to explicitly control how the user agent behaves at these small gaps. BUG=807793 Change-Id: I020e244c230756eaa1804f81b58a577124a6a28b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1428601 Commit-Queue: Matthew Wolenetz <wolenetzchromium.org> Reviewed-by: Dan Sanders <sandersdchromium.org> Cr-Commit-Position: refs/heads/master{#643473} -- wpt-commits: 27ad6759d421b95b4572f20cabaeb750b3eb9799 wpt-pr: 15991 UltraBlame original commit: c679de39ad4dbaafeee76985263d5c11c07de772
Per whatwg/html#6359, we'll likely need to help UAs and apps understand and customize audio vs video-specific playback behavior on network underflow (buffered range gap in each track, here) as part of this solution. |
Briefly discussed during 8 Nov 2022 Media WG meeting: https://www.w3.org/2022/11/08-mediawg-minutes.html#t05 |
This came up again at FOMS2023. It's absolutely a valid issue that still needs to be solved.
Yes, this is ugly: non-interoperable. Not guaranteed to continue working. Adds overhead to time-to-first-frame (async buffering and removal). |
Thanks for this, really interesting. I would like to share my observations: Test setupI created a media segment with a single IDR frame and a duration of 4 seconds. Note that ![]() Problem encounteredNow simply appending the corresponding init segment and the above media segment to the SourceBuffer leads to a buffered range from 0 - 4 as expected. But when removing the buffered range and hitting "play" the current time of the video element is still progressing although there is no valid buffered range: ![]() As a comparison, Firefox is behaving as expected: ![]() I also did not manage to replace the buffered IDR frame with new content. The new content only becomes visible after the segment with the IDR frame has finished (after 4 seconds) ![]() ![]() SolutionIt looks like a simple seek (e.g. seeking to 0.1) solves the previously described problem: ![]() As @wolenetz described Chrome is playing over gaps in my tests afterwards. In this example, there was a gap between 2-4 seconds: ![]() For comparison Firefox stalls after 2 seconds: ![]() |
I put my findings in a blog post: https://websites.fraunhofer.de/video-dev/being-trapped-in-a-gap-with-big-buck-bunny/ : Happy to get feedback, maybe I also missed something in my implementation and the gap workaround can be used without a seek. |
@dsilhavy Nice write-up! It looks like the "ugly hack" is working for you for video. Seek is understandably a "fix" for the Chromium media pipeline having started prerolling the decoder with the (later removed) media. In this case, the prerolling is aborted by a seek to a time that is not currentTime. Without the seek / preroll abort, the Chrome media pipeline will still accept the newly buffered content into the timeline, but the renderer will throw away decode output with earlier presentation time than what was expected to follow the prerolled content. At least, I think this is what is happening. To avoid needing to seek, try buffering the fake content elsewhere in the timeline (say, far in the future), then remove it, optionally adjust the MediaSource duration, and then append the real content. How far in the future? This is another point of lack of interop precision. I'd suggest more than thrice the manufactured frame's duration, and no less than 10 seconds, to hopefully avoid prerolling any decode of the fake content. While considering the "ugly hack" further, I recalled more ugly details. The hack will likely lead to issues even now in Chromium when attempting to auto-coalesce gaps in an audio track's buffered ranges for manifold reasons:
There are likely other caveats, further motivating getting this spec issue fixed and implemented. A couple other notes:
|
In order to support use cases such as trick play, as well as very low latency playback, it may sometimes be useful to be able to allow the media time to keep incrementing at all costs whether or not there is any media to be played.
Ways in which I can imagine this working could be:
a) signal a SourceBuffer as potentially containing sparse data and not to stall the timeline during unbuffered ranges (exact behaviour to be defined eg hold last frame)
b) explicitly 'append' an 'empty' range
I appreciate that this may be better solved by the media element, or at application level, but I think it is at least worth considering in MSE.
I intend this is to be considered a VNext use case.
The text was updated successfully, but these errors were encountered: