Should rendering behavior of the remote-d media element be specified?

[mfoltzgoogle]

I didn't see any language in the spec that the user agent initiating remote playback is required to stop rendering media in the element after connection to a remote playback device. But I would assume this behavior is intended, unless there is a use case for simultaneous playback (i.e., connection to a recording device or showing the same media in two rooms).

[avayvod]

@foolip for guidance. I would imagine that unless this is mirroring it would be very hard for the user agent to maintain the externally visible player state and both local and remote playback states in sync. I'm fine with leaving that to the implementation with some guidance (something like "stopping local playback is recommended to save battery and provide better user experience").

[mounirlamouri]

I believe Anton's WIP draft mentions a local playback state and an external playback state. Shouldn't that address your issue Mark?

[foolip]

The main thing that comes to mind is what happens to the videoWidth and videoHeight attributes, and what happens if the video size changes on the remote side. In other words, should the local layout depend on what happens on the remote side, should it be frozen at connection time, or should the intrinsic size of the video element revert back to 300x150?

[mounirlamouri]

It sounds to me that freezing the layout to whatever it was before the connection is the best way to go. It would make transitioning back and forth from local to remote playback smoother.

[foolip]

Can you check what Safari does with its prefixed API? If it's something sensible then just matching that would be good.

Spec-wise, the concept that needs to be amended is the list after "A video element represents what is given for the first matching condition in the list below" in https://html.spec.whatwg.org/multipage/embedded-content.html#the-video-element

[mfoltzgoogle]

@mounirlamouri @avayvod Have you had a chance to check the Safari behavior?

Re: #46 (comment) I don't think the local playback state completely covers the original question raised by this issue.

As media element state is currently defined in the spec it covers HTMLMediaElement properties but does not address the rendered output of the element, so it leaves it up to the implementer to implement remoting or mirroring. I think a note as suggested by @avayvod would address this.

Second, media element state doesn't cover HTMLMediaElement ancestor properties relating to layout, style, etc. so there's no requirement to synchronize them to the remote playback. I'm not sure freezing layout is the best plan, since the controlling page might want to minimize, hide, or otherwise restyle the local video element during remote playback. Obviously the site would need to restore previous style if remoting is ended to transition back to good local playback experience. Does Safari allow restyling of remoted media elements?

[avayvod]

@mfoltzgoogle I think we're not discussing freezing the element's layout, but the videoWidth and videoHeight properties. That way if the page doesn't do anything to resize the remotely played element, it won't break because the UA suddenly changed the size of the video to say full HD.
The page can still detect if remote playback is happening and resize the element any way it wants.

The rendering should be up to the UA:

• show some default remote playback UI
• show the last rendered frame / poster image
• keep rendering the media as is
  Some things like closed captions may continue being rendered within the media element, for instance, if the remote playback device doesn't support it.

We should advise the UAs to suspend the local playback except for the parts that are needed for remote playback implementation to work (e.g. mirroring/remoting) to preserve resources, esp. if the page will show its own UI on top or instead of the element during remote playback.

[mfoltzgoogle]

That way if the page doesn't do anything to resize the remotely played element, it won't break because the UA suddenly changed the size of the video to say full HD.

I guess while remoting the tradeoff is between:

• Freezing videoWidth and videoHeight and breaking some content (and the spec) when the intrinsic dimensions of the video don't match
• Not freezing them and breaking some content when these are changed by the UA by remote playback

I feel like the latter is less risky: if the media is offered in multiple resolutions, then the page should expect video dimensions to change and the layout should have constraints set to accommodate. But, I don't have concrete evidence to support that claim.

We should advise the UAs to suspend the local playback except for the parts that are needed for remote playback implementation to work

If we do that, then the third bullet you propose is off the table? Or is it possible, but you'd advise against it?

[mfoltzgoogle]

A short experiment playing a 1080p video [1] showed that (videoWidth,videoHeight) remain at (1920,1080) during AirPlay on a 720p Apple TV. This was with Safari 10.1.

[1] http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov

[chrisn]

The rendering should be up to the UA:

• show some default remote playback UI
• show the last rendered frame / poster image
• keep rendering the media as is
  Some things like closed captions may continue being rendered within the media > element, for instance, if the remote playback device doesn't support it.

It would seem useful to allow page authors some control over these choices, rather than leaving it a purely UA decision.

[avayvod]

@chrisn I think that's already possible: websites already can hide or cover the media element with any UI they want as long as they know it's in a connected state.

[mfoltzgoogle]

That's true, but if there's inconsistency between user agents it encourages web developers to hide the media element during remote playback to ensure a consistent experience, which could impact e.g. rendering of captions.

[anssiko]

if there's inconsistency between user agents it encourages web developers to hide the media element during remote playback to ensure a consistent experience

Adding to that point:

Web developers have been hiding the <input type=file> UI for ages and reimplemented it themselves for the same reason, see e.g. the IE5-era technique, a more modern technique, another example. IIRC restrictions to styling the said element were put in place for security reasons to mitigate attack where the user is tricked into opening the file chooser.

Perhaps worth considering this viewpoint when we do decisions on this API. In particular, this seems to confirm web developers go the extra mile to create a consistent experience if needed. I feel they shouldn't be forced to without a good reason.

[mounirlamouri]

TPAC F2F resolution: add note to spec with regards to UI recommendations.

[mfoltzgoogle]

I believe the PR covers the consensus on this issue in the most recent comments.

AddressingvideoWidth and videoHeight properties can be done with implementation experience / developer feedback on how media element state is handled (per the resolution to Issue #41).