Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sequence AppendMode can produce surprisingly bad results with muxed media #186

Open
wolenetz opened this issue Jun 24, 2017 · 8 comments

Comments

@wolenetz
Copy link
Contributor

commented Jun 24, 2017

I think various co-editors and web authors over time, including myself, have noticed problems with spec-compliant processing of muxed coded frames in sequence AppendMode. The issues I see usually stem from the auto-updated timestampOffset, calculated on the first frame processed after a discontinuity in one track, becomes applied to all of the tracks. This can result in loss of A/V sync. There are other cases that, depending on implementation choices, can result in buffered range gaps and playback stalls, too.

Though I recall we had chatted about possibly deprecating/disallowing 'sequence' mode usage for SourceBuffers that have more than 1 track, I couldn't find a resulting spec bug. Let's please discuss here.

Thanks!

@jyavenard

This comment has been minimized.

Copy link

commented Jun 24, 2017

I support this...

In fact it could be argued that there are many ways the current spec could be interpreted when there are multiple tracks.

MXEBot pushed a commit to mirror/chromium that referenced this issue Jun 29, 2017
Due to spec-compliant processing of muxed sequence mode appends
producing problematic results following discontinuities in the muxed
media, this change adds a warning to chrome://media-internals if such an
append is detected. I've also revived spec discussion on this by filing
w3c/media-source#186

BUG=728477

Change-Id: I78858ff9c0982eb76bcdb907a7a3fcfdb5d36b87
Reviewed-on: https://chromium-review.googlesource.com/546963
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Reviewed-by: Chrome Cunningham <chcunningham@chromium.org>
Cr-Commit-Position: refs/heads/master@{#483240}
@wolenetz

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2017

So far, preliminary (Chrome M61 - dev channel) shows very miniscule % of occurrences of sequence muxed mode usage. However, this data is very preliminary, I'll update with public numbers once the related telemetry logging is in a more significant portion of Chrome usage.

@guest271314

This comment has been minimized.

Copy link

commented Sep 17, 2017

Note, not sure about the exacting technical intent or correct interpretation of the specification, here.

As a user of Chromium and Firefox, the implementations of each browser differ significantly enough that adjustments to code to solve an issue at one browser in this case https://bugzilla.mozilla.org/show_bug.cgi?id=1400587 exposed an issue at the browser where code previously rendered expected result.

@wolenetz

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2017

Lack of interop among implementations (and among developer expectations of what should happen following discontinuities in multi-track 'sequence' mode) is precisely why this form of support is strongly being considered to be deprecated.

@Codeusa

This comment has been minimized.

Copy link

commented Sep 18, 2017

@wolenetz to chime in, we currently use sequence mode with multiple tracks to achieve real-time < 20MS streaming. The issue we see with segment (and this could be an miss implementation on our end) is that if a single buffer is missed the playback stops. This is problematic when using WebRTC unreliable channels to pipe the video feed. It seems when using segment mode there is an inherit latency as well making remote desktop applications unusable.

I hope at some point the abstraction layer of MSE is lifted and we can access the decoder ourselves, but until then we're working within the confines of the API. Any tips are welcomed.

Of course if this is deprecated the obvious solution is to split the audio/video into their own streams, its just not as clean.

@jyavenard

This comment has been minimized.

Copy link

commented Sep 19, 2017

@Codeusa that's likely due to a gap in your data. You can verify this by checking the buffered range. If the gaps are too big, playback will stop (as per spec, as you can only play content when readyState is HAVE_FUTURE_DATA).

That's one downside of MSE, it doesn't cope with missing data.
We hope to address this with a "live" MSE mode that would allow for such issue.

in the mean time, you must monitor the buffered range as well as the waiting event, and seek accordingly over the gap (this is what most DASH players are already doing)

@Codeusa

This comment has been minimized.

Copy link

commented Sep 19, 2017

@jyavenard Do you have a particular link where I can follow the progress of the "live" mode spec?

@wolenetz

This comment has been minimized.

Copy link
Contributor Author

commented Sep 19, 2017

@Codeusa , we have several related MSE spec issues around:

  • an API to let apps request for the engine to play through unbuffered regions of the timeline (#160)

  • a (slightly) more explicit low-latency API (not a decoder API) (#21 and #133)

Along with some other highly asked for items, these are likely candidates for upcoming feature incubations as we begin gaining traction on vNext features.

xqq pushed a commit to xqq/chromium-media that referenced this issue Nov 14, 2017
Due to spec-compliant processing of muxed sequence mode appends
producing problematic results following discontinuities in the muxed
media, this change adds a warning to chrome://media-internals if such an
append is detected. I've also revived spec discussion on this by filing
w3c/media-source#186

BUG=728477

Change-Id: I78858ff9c0982eb76bcdb907a7a3fcfdb5d36b87
Reviewed-on: https://chromium-review.googlesource.com/546963
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Reviewed-by: Chrome Cunningham <chcunningham@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#483240}
Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src
Cr-Mirrored-Commit: 3a083529baf8043e94aa394b810acdc320d953c8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.