Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should rendering behavior of the remote-d media element be specified? #46

Closed
mfoltzgoogle opened this issue Jun 30, 2016 · 15 comments

Comments

@mfoltzgoogle
Copy link
Contributor

commented Jun 30, 2016

I didn't see any language in the spec that the user agent initiating remote playback is required to stop rendering media in the element after connection to a remote playback device. But I would assume this behavior is intended, unless there is a use case for simultaneous playback (i.e., connection to a recording device or showing the same media in two rooms).

@avayvod

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2016

@foolip for guidance. I would imagine that unless this is mirroring it would be very hard for the user agent to maintain the externally visible player state and both local and remote playback states in sync. I'm fine with leaving that to the implementation with some guidance (something like "stopping local playback is recommended to save battery and provide better user experience").

@mounirlamouri

This comment has been minimized.

Copy link
Member

commented Jul 6, 2016

I believe Anton's WIP draft mentions a local playback state and an external playback state. Shouldn't that address your issue Mark?

@foolip

This comment has been minimized.

Copy link
Member

commented Jul 6, 2016

The main thing that comes to mind is what happens to the videoWidth and videoHeight attributes, and what happens if the video size changes on the remote side. In other words, should the local layout depend on what happens on the remote side, should it be frozen at connection time, or should the intrinsic size of the video element revert back to 300x150?

@mounirlamouri

This comment has been minimized.

Copy link
Member

commented Jul 6, 2016

It sounds to me that freezing the layout to whatever it was before the connection is the best way to go. It would make transitioning back and forth from local to remote playback smoother.

@foolip

This comment has been minimized.

Copy link
Member

commented Jul 6, 2016

Can you check what Safari does with its prefixed API? If it's something sensible then just matching that would be good.

Spec-wise, the concept that needs to be amended is the list after "A video element represents what is given for the first matching condition in the list below" in https://html.spec.whatwg.org/multipage/embedded-content.html#the-video-element

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2017

@mounirlamouri @avayvod Have you had a chance to check the Safari behavior?

Re: #46 (comment) I don't think the local playback state completely covers the original question raised by this issue.

As media element state is currently defined in the spec it covers HTMLMediaElement properties but does not address the rendered output of the element, so it leaves it up to the implementer to implement remoting or mirroring. I think a note as suggested by @avayvod would address this.

Second, media element state doesn't cover HTMLMediaElement ancestor properties relating to layout, style, etc. so there's no requirement to synchronize them to the remote playback. I'm not sure freezing layout is the best plan, since the controlling page might want to minimize, hide, or otherwise restyle the local video element during remote playback. Obviously the site would need to restore previous style if remoting is ended to transition back to good local playback experience. Does Safari allow restyling of remoted media elements?

@avayvod

This comment has been minimized.

Copy link
Contributor

commented May 15, 2017

@mfoltzgoogle I think we're not discussing freezing the element's layout, but the videoWidth and videoHeight properties. That way if the page doesn't do anything to resize the remotely played element, it won't break because the UA suddenly changed the size of the video to say full HD.
The page can still detect if remote playback is happening and resize the element any way it wants.

The rendering should be up to the UA:

  • show some default remote playback UI
  • show the last rendered frame / poster image
  • keep rendering the media as is
    Some things like closed captions may continue being rendered within the media element, for instance, if the remote playback device doesn't support it.

We should advise the UAs to suspend the local playback except for the parts that are needed for remote playback implementation to work (e.g. mirroring/remoting) to preserve resources, esp. if the page will show its own UI on top or instead of the element during remote playback.

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2017

That way if the page doesn't do anything to resize the remotely played element, it won't break because the UA suddenly changed the size of the video to say full HD.

I guess while remoting the tradeoff is between:

  • Freezing videoWidth and videoHeight and breaking some content (and the spec) when the intrinsic dimensions of the video don't match
  • Not freezing them and breaking some content when these are changed by the UA by remote playback

I feel like the latter is less risky: if the media is offered in multiple resolutions, then the page should expect video dimensions to change and the layout should have constraints set to accommodate. But, I don't have concrete evidence to support that claim.

We should advise the UAs to suspend the local playback except for the parts that are needed for remote playback implementation to work

If we do that, then the third bullet you propose is off the table? Or is it possible, but you'd advise against it?

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2017

A short experiment playing a 1080p video [1] showed that (videoWidth,videoHeight) remain at (1920,1080) during AirPlay on a 720p Apple TV. This was with Safari 10.1.

[1] http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov

@chrisn

This comment has been minimized.

Copy link
Member

commented May 17, 2017

The rendering should be up to the UA:

  • show some default remote playback UI
  • show the last rendered frame / poster image
  • keep rendering the media as is
    Some things like closed captions may continue being rendered within the media > element, for instance, if the remote playback device doesn't support it.

It would seem useful to allow page authors some control over these choices, rather than leaving it a purely UA decision.

@avayvod

This comment has been minimized.

Copy link
Contributor

commented May 17, 2017

@chrisn I think that's already possible: websites already can hide or cover the media element with any UI they want as long as they know it's in a connected state.

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor Author

commented May 19, 2017

That's true, but if there's inconsistency between user agents it encourages web developers to hide the media element during remote playback to ensure a consistent experience, which could impact e.g. rendering of captions.

@anssiko

This comment has been minimized.

Copy link
Member

commented Oct 24, 2017

if there's inconsistency between user agents it encourages web developers to hide the media element during remote playback to ensure a consistent experience

Adding to that point:

Web developers have been hiding the <input type=file> UI for ages and reimplemented it themselves for the same reason, see e.g. the IE5-era technique, a more modern technique, another example. IIRC restrictions to styling the said element were put in place for security reasons to mitigate attack where the user is tricked into opening the file chooser.

Perhaps worth considering this viewpoint when we do decisions on this API. In particular, this seems to confirm web developers go the extra mile to create a consistent experience if needed. I feel they shouldn't be forced to without a good reason.

@mfoltzgoogle mfoltzgoogle added the F2F label Nov 2, 2017
@mounirlamouri mounirlamouri self-assigned this Nov 6, 2017
@mounirlamouri

This comment has been minimized.

Copy link
Member

commented Nov 6, 2017

TPAC F2F resolution: add note to spec with regards to UI recommendations.

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor Author

commented Jan 11, 2018

I believe the PR covers the consensus on this issue in the most recent comments.

AddressingvideoWidth and videoHeight properties can be done with implementation experience / developer feedback on how media element state is handled (per the resolution to Issue #41).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.