-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to handle changes to the set of tracks in a MediaStream assigned to an element via the srcObject attribute #453
Comments
Thanks @guidou for starting to test this. I did write some of the spec parts, but I do think there may be gaps and unclear things there (not to mention that the media element and the track lists seemed to move a bit as well). My intention at the time of writing was that adding and removing tracks would be equivalent with how it works if a file is allocated (using |
Good that you bring this up @guidou. When doing some work in this area for Gecko I found the most sensible approach was to:
That said, I am not sure Gecko's implementation fully follows this, though it's probably close. It does help that we don't support AudioTrack and VideoTrack (for now prefed off), or we'd have to deal with issues such as:
The only text I find on mutating the TrackLists is on when a track ends, but that doesn't mention whether another track should be enabled or selected. Perhaps I missed something? @stefhak mentions tracks coming and going when a media element plays a file but to my understanding all tracks of a file src are known when metadata has been loaded, and this is the only case that the spec supports. I believe that MSE supports tracks that come and go however, so it would be interesting to know how it works there. |
The MSE spec allows some flexibility.
In case user agent allows new SourceBuffer or track to be added/removed, it'd make sense for the user agent to go through the load algorithm and trigger corresponding events. Apps can listen to the events and update the app UI accordingly. This seems a good topic to bring to the MSE folks. It'd be ideal for MSE and MediaStream objects to have consistent behavior rather than making separate assumptions. |
In WebKit, there should be no difference between MSE, gum, and peer connection media streams. |
That sounds the right behavior given the user agent will fire loadedmetadata and loadeddata events, etc. due to the track change. |
@Pehrsons : I was thinking about the text saying Anyway, that text should IMO be valid also if the source is a |
Ah, it does actually mention tracks being added or removed dynamically. That's good. However, I haven't found any language on how to handle such events.
I don't know whether or not this is a strict requirement (it appears not, per above), but for playing a simple file this is true in practice - from parsing container metadata. |
@ShijunS good points in #453 (comment). We should probably have something similar (i.e. allow the html media element to throw if it can't handle one more track). That though brings up the question if we should have similar errors on recorder, peer-connection etc. |
I ran a test with Firefox, Safari and Chrome to see how they're handling this nowadays. The test consists in the following steps:
Firefox and Chrome behave the same. They don't run the full load algorithm and fire the "resize", "canplay" and "playing" events on step 2. They don't fire any events on step 3 and 4. If the track added in step 4 has a different size, "resize" is fired. Safari fires "resize", "loadedmetadata" and "canplay" on step 2; fires "resize" on step 3; and fires "resize", "loadedmetadata" and "canplay" again on step 4. I guess the reason for the resize events in step 3 and 4 for Safari is that it internally changes the size on track removal while Chrome and Firefox do not. Given that Firefox and Chrome coincide and that seems to match the media element spec, I think we should considered that behavior as correct. Not sure if that requires changes to the latest version of the spec. |
This seems wrong. The media element spec says the 'resize' event should be fired:
I would think that "the selected video track was changed" includes removing an active video track.
Removing the video track shouldn't cause the readyState to drop below HAVE_METADATA, so firing 'loadedmetadata' and 'canplay' also seems wrong. I filed https://bugs.webkit.org/show_bug.cgi?id=225612 to investigate this. |
The spec says:
However, the exact mechanism for this is not explained.
I tested adding and removing tracks to a stream with Edge, and it apparently runs the load algorithm every time the set of tracks changes, since it fires the emptied and loadstart events on track addition/removal.
On Firefox, adding or removing a track results in it being rendered by the element, but the full load algorithm does not run, since loadstart/emptied are not fired.
On Chromium, both adding and removing a track are ignored by the element. I was actually trying to fix this bug on Chromium and initially went the Edge way, but @foolip suggested that the Firefox approach might be more in line with the way the media element is supposed to work.
Should a specific approach be specified here, or are both approaches equally valid?
The text was updated successfully, but these errors were encountered: