-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clarify how track buffer ranges are updated. #15
Comments
We've looked at this some more, and think we might want to reconsider putting this on V.Next. The issue looks like an edge case that would not be encountered in normal streaming media, as in the long frame display example. It seems like a fit on V1 only if a clear change can be defined for it soon. |
There were multiple issues identified in the previous w3c bug that still need to be broken out into distinct spec issues (which I'll do as part of follow-up). See especially acolwell's response in https://www.w3.org/Bugs/Public/show_bug.cgi?id=27242#c5. With respect to the original issue (which demonstrated lack of spec clarity around what should buffered show for IPBBB.. when not all of the B frames have been appended yet), I believe a much of the direction was provided in https://www.w3.org/Bugs/Public/show_bug.cgi?id=27242#c2. @jdsmith3000, to which example were you referring to with "the long frame display example"? |
Action on me: split this into multiple (one covering the original, with direction of fix outlined in bugzilla's comment 2, and multiple outlined in bugzilla's comment 5.) |
We've run into a scenario that I don't think is completely captured in the bugzilla issue but seems very closely related so I'll record it here. The current extensions to HTMLMediaElement.buffered specify the buffered attribute return the intersection of the buffered TimeRanges of all active SourceBuffers when a MediaSource is being used. In some cases, this doesn't align with the goal behavior described in the bugzilla issue:
Given the sampling rates for audio and video are different, it's often the case that audio starts a short while after the video, or vice versa. In those cases, the user agent reports a non-zero starting time for the first buffered time range at the Media Element level, even when that gap would be happily ignored for the purposes of playback. This can cause confusion for application code that attempts to be smart and backfill gaps that appear in the buffer. |
I believe at least some minimum clarification for how a jagged-starting stream's (audio and video tracks starting slightly offset from each other) buffered TimeRanges are calculated for each of two cases:
I believe addressing these specific concerns is appropriately scoped to V1, and does not require breaking this out into multiple issues (in V1 time-frame). |
For clarity, I think this falls into V1 per step 6 of the triage process currently. |
#15 (comment) tracks the approach planned for solving this issue. In addition, should we consider formalizing what to do in the following case: ? |
@jdsmith3000 friendly ping :) |
I need to confirm details of this with our Media team. I've discussed it with them and they've committed to a response by Friday. I'll comment again then. |
Thanks @jdsmith3000 . I look forward to a response Friday so we can get this fixed ASAP. |
It appears these issues arise because MSE cannot describe the buffered ranges accurately for a sourceBuffer with muxed AV content because there is a single buffered range for the sourceBuffer even though it may contain multiple tracks. In practice, if apps are using the buffered range as an indicator of fullness to control buffering logic, this is not a problem. However, if apps are trying to perform frame accurate edits, issues like these will occur. That's what's behind my previous comment that these issues do not seem like V1 topics. On the three issues you highlight:
Also, since “sequence” mode is mentioned, here are some thoughts on it. “Sequence” append mode doesn't work well for muxed AV content in a single sourceBuffer due to the same issues described above, and the fact that there is only one timestampOffset that would get applied to all tracks even though appending a single MEDIA segment will result in slightly different durations appended for each track within the sourceBuffer. Audio and video durations just do not align precisely. That shouldn't matter in practice. “Sequence” append mode should ideally not be used for muxed AV content, and should only be used for appending elementary audio streams. We might want to consider limiting “sequence” append mode to audio elementary stream formats. |
@wolenetz: Jerry has provided his input here. What are the next steps here? |
@paulbrucecotton I'll produce a PR shortly. I think @jdsmith3000's response to my #15 (comment) is pretty much what I was thinking. Regarding the 'sequence' appendMode clarifications, those are out of scope of this issue, and I'm not sure they need clarification in the MSE spec. Note that 'sequence' appendMode relies on the same discontinuity detection logic as 'segments' appendMode, though 'sequence' uses it to adjust timestampOffset. In 'sequence' appendMode, certainly there could be problems with muxed AV, especially audio, where it might not play "continuously" and align with the resequenced timestamps since there is allowance in the discontinuity detection logic for some small gap. Likewise, I've seen 'sequence' appendMode most applicable in single-stream SourceBuffers, especially audio, for things like gapless playback. In fact, audio/aac and audio/mpeg MSE bytestreams require 'sequence' mode behavior. @jdsmith3000 perhaps clarifying with some non-normative note our thoughts/warnings about the probably playback experience for muxed AV in 'sequence' appendMode might qualify as a distinct V1NonBlocking spec issue : please file one if you agree. |
Have been able to use |
Migrated from w3c bugzilla tracker. For history prior to migration, please see:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=27242
The text was updated successfully, but these errors were encountered: