Presenting the content of an <audio> or <video> element

[mfoltzgoogle]

From, e.g. http://lists.w3.org/Archives/Public/public-webscreens/2014May/0061.html and replies:

Requesting display of non HTML content

---

The second issue at stake is about specifying what happens when a call such as the following is issued:

navigator.presentation.requestShow('video.mp4');

The main problem is that, regardless of where this runs, this does not create a "browsing context" as such since a video is not an HTML document.

While it's easy to see how a browser could create a fake browsing context for the video (as is done in practice when you open a link to a video in your browser), it's hard to see how the initiating page can communicate with the created browsing context to control video playback.

The proposed solution would involve the

This issue tracks the work needed to specify this behavior.

[avayvod]

Thanks Anssi for tagging it with F2F. I'd like to see this discussed after the higher priority Presentation API issues. I drafted a change proposal to the HTMLMediaElement and HTMLSourceElement tags that goes like this:

// A twin of AvailableChange event for HTMLMediaElement:
[Constructor(DOMString type,
    optional RemotePlaybackAvailabilityChangeEventInit eventInitDict)]
interface RemotePlaybackAvailabilityChangeEvent : Event {
  readonly attribute boolean available;
};

dictionary RemotePlaybackAvailabilityChangeEventInit : EventInit {
  boolean available;
};

// Event that notifies the page when the media element is being played remotely,
// contains the human readable screen name for better UI integration.
[Constructor(DOMString type,
    optional RemoteStateChangeEventInit eventInitDict)]
interface RemoteStateChangeEvent : Event {
  readonly attribute PresentationSessionState state;
  readonly attribute DOMString screenName;
};

dictionary RemoteStateChangeEventInit : EventInit {
  PresentationSessionState state;
  DOMString screenName;
};

// Two extra methods and two extra events for the HTMLMediaElement
partial interface HTMLMediaElement : HTMLMediaElement {
  // allow the page to be notified if it should show remote playback UI
  // similar to the presentation API, we don't need a matching boolean
  attribute EventHandler onremoteplaybackavailabilitychange;

  // notifies the page when the media starts playing remotely or
  // disconnects; also called if the browser initiates the remote playback
  attribute EventHandler onremotestatechange;

  // starts the remote playback from the current position; UA shows the device
  // picker
  void startRemotePlayback();

  // allows the page to stop the playback, local playback may continue from the
  // current position
  void stopRemotePlayback();
}

partial interface HTMLSourceElement {
  // if true (by default) the src attribute can be used for remote playback; 
  // else, playing this source remotely is disabled.
  attribute boolean allowRemotePlayback=true;
}

This could be polyfilled using the Presentation API provided some defautl media player presentation URL.
I'd like the media to be playable remotely by default but allow the page to mark elements as local only (i.e. for licensing, access control or quality issues). HTMLMediaElement commands like play()/pause() coud be reused to work with the remote playback in the case of playing the media on the second screen so there's no need to invent any messaging protocol.

Some rules will have to be specified regarding CORS, EME, MSE, etc.

Let me know what you think.

[anssiko]

@avayvod Thanks for putting this proposal together. You may want to add couple of examples to demonstrate the common use cases.

I've experimented with a similar idea before (https://github.com/webscreens/requestshowmedia) and personally think sharing video on remote screens is one of the most common use cases. Providing an API optimised for that would make sense.

Depending on how much effort you can put into this, you could even prepare its own Editor's Draft for the feature to ease the review. But that is certainly not required in order to have a productive discussion at F2F.

[avayvod]

Thanks Anssi! I'll look into that.

[foolip]

Yeah, HTMLMediaElement seems to be the appropriate place for this. Note that HTMLSourceElement is intended to pick between different encodings of the same video, so allowing or disallowing remote playback per source seems a bit odd.

Note that the Fullscreen API started as a video-specific thing and then got generalized to all elements. Might a similar thing happen here, so that a Element.requestRemotePresentation() or something would be appropriate? Or possibly it could be integrated with the Fullscreen API with extra arguments to Element.requestFullscreen()? (Sorry if this is nonsensical, I'm not really up to speed with the Presentation API.)

[foolip]

My main interest when it comes to remote media playback is that we can make text track (WebVTT) rendering work. That's why using HTMLMediaElement is appropriate, but it's of course very easy to spec or implement it such that it doesn't actually work.

[drott]

Might a similar thing happen here, so that a Element.requestRemotePresentation() or something would be appropriate? Or possibly it could be integrated with the Fullscreen API with extra arguments to Element.requestFullscreen()? (Sorry if this is nonsensical, I'm not really up to speed with the Presentation API.)

We discussed this at some point, but there are some tricky issues with extracting arbitrary elements from the DOM, then moving individual block-displayed elements and stretching them to the remote screen's aspect ratio etc. What happens to media queries, what happens to associated resources from the original DOM - how do they get transfered to the remote screen? This is hard to implement in a two-user-agents scenario, etc. - that's why we went for passing only a "presentation URL" to the presentation display and using that. So I wouldn't expect an Element.requestRemotePresentation() for elements generically, but for video elements I think this may work.

[foolip]

Yeah, that is hard. I expect that getting text track rendering right is also hard for similar reasons, although one could of course just construct an equivalent DOM on the other side where any URLs have been made absolute and so forth. Not sure how robust that would be, though.

[tidoust]

I note that the approach proposed is somewhat similar to that defined in the Audio Output Devices API specification, produced by the Media Capture task force, joint effort between the Device APIs Working Group and the WebRTC Working Group.

The Audio Output Devices API proposes to add a setSinkId method to MediaElement to associate the element with some media output device (audio-only for the time being) returned from an enumeration of media output devices currently obtained through a call to getUserMedia (but my understanding is that there will be a way to enumerate devices without calling getUserMedia in the end).

I think there have been discussions on filtering media devices based on capabilities, that could perhaps address #9.

It might be useful to investigate whether both approaches could be merged into one. At a minimum, we should get in touch with the Media Capture task force once we have a better idea as to what we are willing to specify to present audio/video content.

[avayvod]

I've updated the proposal and have put it onto my github: https://github.com/avayvod/remotehtmlmedia/blob/master/proposal.md

[anssiko]

ACTION: @avayvod to figure out a name for the spec and continue the work [recorded in http://www.w3.org/2015/05/20-webscreens-minutes.html#action08]

ACTION: @tidoust to create the repo on GitHub for the spec once we have a name (and short name) for it. [recorded in http://www.w3.org/2015/05/20-webscreens-minutes.html#action09]

[duanyao]

I don't think limiting remote presenting to media elements is a good idea. E.g. what if one want to use a video player with custom UI?

This is my proposal (was posted to the mail list "extend fullscreen API to support second screen" ):

Local UA                                   network                              Remote UA

a Element                  -----------captured live video stream------->    a HTMLVideoElement
(requestSecondScreen(),    <----------input events----------------------    (fullscreened, RDP client)
RDP server)
     |
     |
     |
::mirror
(pseudo-element,
a local RDP client)

To summarize:

• Any local element can requestSecondScreen() (a remote version of requestFullscreen()). If succeed, the element is projected to the remote UA.
• The local element projected acts as a RDP server, while the remote UA create a special HTMLVideoElement which acts as a RDP client, and connects to the RDP server. (Not saying that we must use RDP, just an example). The projected element is rendered locally and offscreen, but according to the dimension of the remote screen. Input events of the second screen are sent back to the local UA to process.
• There is a pseudo-element ::mirror acts as a local RDP client in place of projected element to show a scaled-down view of it. ::mirror has naturalWidth|naturalHeight assigned with the dimension of the projected element just before calling requestSecondScreen().
• No cross-document messaging required, and client code just feels it is working with a single document.

[avayvod]

Note that if one wants custom UI for a video player, it's usually still done with the media element. My spec proposal has video/audio player custom UI as the main use case (with the default UI, browser doesn't need an API that much). There's no messaging either, media element methods work on the second screen.

For an arbitrary element it seems much harder to define how it would behave in sync with its own copy that has different dimensions. What would properties like width and height return, for example?

[duanyao]

Note that a "video player with custom UI" can mean arbitrary element. E.g. a player may show a video inside another video.

For a requestSecondScreen()-ed element, its dimention should be the same as the second screen, similar to the case of requestFullscreen(). The ::mirror pseudo-element has the original dimention of the requestSecondScreen()-ed element.

[mounirlamouri]

This is something Anton and I are really excited about but it shouldn't be part of the Presentation API. Maybe it could be part of the same WG though. Anton moved his draft in its own repository: https://github.com/avayvod/remotehtmlmedia We will iterate from there.

[anssiko]

This was discussed at the Berlin F2F, and it was concluded this feature would be part of the WG scope, but will be worked on in its own spec.

When @avayvod et al. feel like the proposal is ready for a wider review this issue should be pinged so that we can initiate a wider review and work out a plan how and whether to move this feature to the Rec Track.

[tidoust]

Also note the feedback from the TAG on this particular idea:

[[
It would may more more sense to "cast" an HTMLVideoElement to a second screen if the ergonomics of the presentation API more closely matched the fullScreen spec--which is something that could be considered.

Today, we are conflating the notion of a "remote control" browsing context (this specification), with the idea of pure presentation of an element (fullscreen API + casting). Both scenarios seem to have merit, but I agree that they tend to get muddied--even in the use-case document.
]]
https://github.com/w3ctag/spec-reviews/blob/master/2015/presentation-api.md

[mfoltzgoogle]

@avayvod and @mounirlamouri would like to reopen this issue and discuss at the upcoming F2F.

[tidoust]

See relevant discussion at TPAC F2F:
http://www.w3.org/2015/10/29-webscreens-minutes.html#item01

RESOLUTION: for issue #13, Anton and Mounir will spec an Editor's Draft that follows the proposal shared with the group so far

[mounirlamouri]

https://w3c.github.io/remote-playback/

I am closing that issue. I think we should move the discussions about this to the remote playback repo.

[anssiko]

I added the Remote Playback API to the group's home page and publication status page to make sure people are able to find the spec easily.

@tidoust Could you (with help from @dontcallmedom perhaps) enable github-notify-ml for w3c/remote-playback using the same configuration we have for w3c/presentation-api so that the public-secondscreen@w3.org mailing list will be automatically notified of relevant activity in the spec repo?

Any further discussion regarding the Remote Playback API should happen in the w3c/remote-playback repo (preferred) or on the public-secondscreen@w3.org mailing list.

[dontcallmedom]

@anssiko @tidoust I've enabled github-notify for w3c/remote-playback

[anssiko]

@dontcallmedom Thanks!