Overview of Media Technologies for the Web updates

[chrisn]

It would be good to update the Overview of Media Technologies for the Web document with some of the latest developments in media related APIs, which include:

• Specifications now under development in the Media WG
• Latest developments in Second Screen CG and WG, eg, Open Screen Protocol
• Media Timed Events and DataCue proposal
• Web Codecs
• Web Transport
• HTMLMediaElement latency hint proposal

Are there other items to add?

[xfq]

What about Audio Device Client in Media orchestration, and/or webrtc-svc in Media distribution?

[tidoust]

I'm working on a pull request which I hope will be ready in the next couple of days. Below is a quick dump of updates per section that I'm planning to make. It includes the specs mentioned above

Media Rendering

• Update status of WebVTT (now a Candidate Recommendation)
• Merge TTML 2 with TTML 1, move to "well-deployed"
• Move IMSC 1.1 profile to "well-deployed"
• Move Open Screen Protocol to "in progress"
• Move CSS Colors Level 4 to "in progress"
• Update Color Management in "not covered"
• Mention video- plans in CSS Media Queries Level 5 to address bi-plane
• Update Extensions to EME in "not covered" and mention those "in progress"
• Talk about DataCue and next generation TextTrackCue as "exploratory"
• Mention Audio Device Client in "exploratory"

Playback Controls

• Mention Input for Workers and Worklets
• Mention WebXR Gamepads Module
• Move Autoplay Policy Detection to work in progress

Media Distribution

• Expand description of Media Capabilities to cover the "can/should" and describe support for color spaces
• Move codec switching feature for MSE to "in progress" (see description in Media Processing)
• Mention WebTransport
• Mention Content Hints and SVC for WebRTC

Media Processing

• Start with MSE / EME
• Mention WebGPU (and drop entry in not covered)
• Mention WebCodecs in exploratory and in "Media Capture Stream with Worker"
• Mention WebAssembly somehow
• Mention HTMLMediaElement latency hint proposal

Media Orchestration

• Move Media Timed Events to "in progress"
• Mention DataCue and next generation TextTrackCue
• Mention Audio Device Client in "exploratory"

Media Capture

• s/Media Recorder API/MediaStream Recording

Media Application Development

• Update Web Media APIs to 2019

[tidoust]

I'm also planning to mention a couple of proposals discussed in the WICG in Media Processing, such as the proposal for a video.requestAnimationFrame and the proposal for an OffscreenVideo

[nigelmegitt]

Worth mentioning the Audio Description profile of TTML2, ADPT? Currently in development/exploratory, but on Rec track by agreement from TTWG. Needs quite a bit of work on it (by me, when I can switch my focus to it).

[tidoust]

I prepared a pull request against the overview document with the changes I mentioned earlier. The changes include the Audio Description profile of TTML2, and a few other specifications that came to mind while working on the document, including Bullet Chatting, specifications around media packaging (Web App Manifest, Web Packaging, Carriage of Web Resource in ISOBMFF), and media processing (Web Neural Network API).

I didn't necessarily followed my initial plan, meaning that some specifications may be listed in other pages in the end. See the pull request for details:
w3c/web-roadmaps#443

[chrisn]

Thanks, @tidoust. The update looks good, and the website is a great resource to see all the related topics gathered in one place.

[tidoust]

I went ahead and merged the pull request after adding a few additional mentions in response to feedback on the pull request, namely EPUB 3.2, Publication Manifest, Audiobooks, and rendering of captions in VR/360° experiences.

The updated Overview of Media Technologies for the Web is available at:
https://w3c.github.io/web-roadmaps/media/

Anything missing? Anything that could be further clarified?

[tidoust]

One more proposed update, see w3c/web-roadmaps#445

The idea is to refactor the "Media application development" section into a "Device interoperability" section to highlight the fact that the section is also aimed at device manufacturers and integrators, and to complete with technologies that can be used to get performance metrics on a running application (specifications developed by the Web Performance Working Group), and detect capabilities of the underlying platform (Media Capabilities, and CSS Media Queries, which are also mentioned in other sections).

The update also mentions a couple of topics that I keep hearing here and there and that are not really covered by ongoing efforts (integration guidelines and performance requirements).

[tidoust]

I went ahead and merged the pull request to ease review of the resulting page, now visible at:
https://w3c.github.io/web-roadmaps/media/apps.html

Happy to incorporate further feedback!

I note that the implementation info for the Performance Timeline specification in the summary table is a bit misleading for the time being. For instance, browsers that support Resource Timing de facto implement at least the PerformanceEntry interface defined in Performance Timeline. I'll see if I can improve that (the problem being that there isn't a lot of info about that spec in status platform sources)

[tidoust]

Following discussions during last IG call, I created a "heat map" page in the Overview of Media Technologies for the Web document that lists all groups mentioned throughout the pages, along with the specifications they are (or were) responsible for, see:

https://w3c.github.io/web-roadmaps/media/groups.html

This page is automatically generated from the contents of the other pages. Given the current backend design and focus on web technologies of these pages, the list basically only contains W3C groups and the WHATWG. It contains ~30 groups and ~90 specs.

The page is a bit raw for the time being, and e.g. does not distinguish groups that are active from those who are no longer around. It does not distinguish either between specifications that have been published as standards, those that are ongoing, and those that have stalled for some reason. This could perhaps be added later on. However, note that this information can typically be inferred by looking at the contents of the pages that mention these groups and specifications!

[tidoust]

Closing this issue as the batch of updates that was discussed has been integrated in the Overview document. That document is now slightly updated and would certainly benefit from another round of updates. I'll look into it but propose to track this in another issue.