RemotePlaybackState enum can become misleading when changing media.src

[jernoble]

The steps for prompt() allow the promise returned to be rejected with NotSupportedErr if the media element cannot support remote playback. This allows, e.g., UAs which support MSE to indicate that a media element backed by a MediaSource cannot be played back remotely, and further allows sites to switch to file-based playback. But a page that changes the media's src/srcObject from a file-based URL to a MediaSource would only get a "disconnect" event, and would be unable to tell whether the user explicitly disconnected the remote playback session or whether the new src is unsupported.

This could be mitigated by adding a new RemotePlaybackState enum, such as (straw man) "unsupported", with a matching "unsupported" and onunsupported message and event handler.

Sites who start loading a media element with a file-based URL, successfully initiate remote playback with prompt(), and then switch the src/srcObject to a MediaSource would (straw man) receive a "unsupported" message which would allow them to switch back to a file-based URL.

[jernoble]

This came up in the context of a recent bug report where YouTube.com is playing a video playlist in Safari while playing back wirelessly to an AppleTV, so this is a real world problem affecting a well-visited media site.

[mfoltzgoogle]

CC @mounirlamouri for his thoughts.

This proposal seems reasonable to me.

Thinking through how resumption would work: if the video gets an unsupported event and switches back to a supported (file-based) URL, it could call prompt() again and the browser could handle that as a special case and resume playback on the same device (if it's still available).

We could add some additional resume() API if that isn't sufficient, but I don't see a need currently.

[mounirlamouri]

Instead of adding a new state, could we add information to the disconnect event that explains why the disconnection happened?

[jernoble]

@mounir said "Instead of adding a new state, could we add information to the disconnect event that explains why the disconnection happened?"

I'm not sure how that simplifies things. "disconnected" doesn't appear to imply that the element will attempt to reconnect when the src changes. Also, in the "Browser initiated remote playback" case, you'd either get a "disconnected" event without a matching "connecting" one, or no events at all, depending on your interpretation of the "Browser initiated remote playback" section.

[mounirlamouri]

My concern is that by introducing a new state, it makes things more confusing. I see the RemotePlaybackState as a connection state and unsupported isn't much of a connection state. Can you go from unsupported to disconnected? or to connecting? Maybe there is a very simple graph that could show how to go from each of these states but after a quick look, it doesn't seem "simple".

I'm not sure how that simplifies things. "disconnected" doesn't appear to imply that the element will attempt to reconnect when the src changes.

Is it what we want though? If we tell the page what happened, shouldn't we let the page decide?

[jernoble]

My concern is that by introducing a new state, it makes things more confusing. I see the RemotePlaybackState as a connection state and unsupported isn't much of a connection state. Can you go from unsupported to disconnected? or to connecting?

To answer your question with another question, can you go from disconnected to disconnected? I don't think that makes sense, but that's what would happen if you used the default media controls to try to initiate remote playback, or if the system/UA tried to initiate Remote Playback. You'd either get a disconnected event without a matching connecting one, or you'd get nothing at all, which is worse.

So, can you go from unsupported to disconnected? Sure, if the user explicitly selects local playback as a result of prompt(), or in a similar interface provided by the UA, or if the remote device becomes unavailable. Can you go from unsupported to connecting? Sure, if the video.src changes to a supported media type, or if the user selects a different endpoint that does support the current media type. I think all that state transition is clear and makes sense.

Is it what we want though? If we tell the page what happened, shouldn't we let the page decide?

As I said above, there's a hole in that flow, namely around UA-driven remote playback. I also think it's weird that every time a page gets a "disconnected" event, they have to check some status code to try and tell if the disconnection is recoverable. Another is that a) video.src = null; video.load(); video.src = 'foo.mp4', b) video.src = 'foo.mp4' and c) video.srcObject = mediaSource; video.src = 'foo.mp4' all have different behaviors. If you start with a connected remote playback session, b) will result in a connected remote playback session, c) will result in a disconnected remote playback session, and a) is not really clear about whether it will or not. Introducing a unsupported state would mean all three would result in the same behavior: "foo.mp4" would start playing remotely.

[mounirlamouri]

I guess there is one key implementation point: I would assume that when receiving data that isn't compatible, the UA would disconnect from the remote screen and wouldn't attempt to auto-reconnect but instead leave it to the website to decide. You seem to suggest that the UA should keep a connection alive and attempt reconnecting as soon as possible. Is that correct?

[jernoble]

This is currently the platform behavior of AirPlay on iOS yes. I'm not saying that the spec should require this behavior, but the spec should allow this behavior to occur.

That said, the only specced mechanism for a website to trigger remote playback is the prompt() function. It doesn't seem like reasonable behavior to present a "remote device picker" to the user, change the element's source, and present the same "picker" again in order to establish a connection. If UAs were given the freedom to allow a re-connect to a previously chosen remote device upon source change, I'm guessing most would do so.

[tidoust]

This was discussed during the Second Screen WG F2F in Fukuoka.

Main options raised during that meeting:

1. Add something like a connected-but-unavailable state. To get from this state to connected, application needs to change src.
2. Use disconnected state and provide a reason. To get back to connected, there should be a reconnection mechanism that does not prompt the user.
3. Stay with connected and fire a new event, noting that applications that don't listen to the new event will assume that everything's going well.

Also the point was made that supporting source switching may actually be optional.

[mounirlamouri]

@jernoble I see that you have the API enabled in a Tech Preview but a look at the WebKit code shows that you haven't added a new event/state. Did you go with one of the solutions above or is this still something to be decided?

[jernoble]

Still to be decided.

[mounirlamouri]

Having a quick look at the spec again, I would like to propose this simple addition:
When the device is in the connecting state (ie. prompt() isn't resolved), if the user agent detects that the remote device doesn't support the source of the local playback, it can fire an event named src-not-supported. If the source of the media element is changed in the event handler, the user agent may attempt to connect again (while keeping the observable state to connecting). The promise should then be resolved and the state updated to either connected or disconnected.

@jernoble WDYT?

[jernoble]

I have a counter-proposal: rather than sending across the current element URL, which may or may not be supported by the remote device, the UA should send over all the sources, their various types, medias and whatnot, and allow the UA to select the appropriate source for itself. Sites could opt in with a declarative change, no new state would be necessary, no new event, and the adoption path much more straightforward.

[mounirlamouri]

To be sure I understand, you are suggesting that the remote device runs the source selection algorithm and be sent all the information it needs instead of having the UA decide what to send? The idea of it sounds fine. I guess the spec could be more vague and just say that the UA and the remote device should pick the right source based on the currently available sources.

@jernoble is my understanding correct? FWIW, I thought that in your case, you would want to swap from MSE to HLS or something like that. I thought that these would use src directly.

@mfoltzgoogle would you have any recommendation based on your work on Open Screen?

[jernoble]

is my understanding correct?

Yes, that's pretty much correct. And you're right that the spec could be pretty vague on this point, and just refer back to HTML to the source selection algorithm. And yes, this would require sites to use multiple elements rather than just src=, but any solution is going to require adoption to see any benefit, and this sounds less risky and error prone then adding a new state or sub-state.

[mounirlamouri]

I'm happy with this approach. I will sync with @mfoltzgoogle and send a pull request this week for review.

[mfoltzgoogle]

From an Open Screen point of view, this sounds fine. The protocol allows multiple media resources to be sent with the remote playback start request, at which point the Open Screen receiver can use its own source selection logic to decide what to play. Here is the relevant section of the protocol spec (paragraph starting with "When section 6.2.4 says...")

Of course, if the user agent isn't using the Open Screen protocol, then it can use whatever mechanism it has to negotiate with the remote playback device which source to play (like knowing that certain devices support certain sources or not).