WebCodecs API

[BearCooder]

Description

The WebCodecs API gives web developers low-level access to the individual frames of a video stream and chunks of audio. It is useful for web applications that require full control over the way media is processed. For example, video or audio editors, and video conferencing.

Source

Read more in this web dev article

Rationale

There are several online editors for editing and processing audio and video files on the internet these days. Such websites eliminate the need for downloading and installing heavy-duty software on your computer. All you need to do is search for online editing tools, and Google will provide you with all the websites providing this service.

From the programmer’s perspective, building such web applications can be difficult. There are a lot of features that they need to implement to make a functional online video editor. And without a doubt, APIs will come into play.

WebCodecs API is a powerful web API that lets developers access the individual frames of a video. It also helps developers to encode and decode audio and video files. It does all this by working on a separate thread. This way, the site responsiveness is not compromised by the processing load.

The API provides access to a lot of features, for instance,

• Raw video frames
  

• Image decoder
  

• Audio/Video encoders and decoders
  

• Audio data
  

The WebCodecs API is under the hood used by several other APIs. For instance, Web Audio API, MediaRecoder API, WebRTC API, etc., all use WebCodecs for processing audio and video.

The WebCodecs API consists of a lot of different interfaces, but almost all of them are not supported by Firefox and Safari.

Specification

W3C

Tests

No response

[foolip]

Tests are here: https://wpt.fyi/results/webcodecs

The tests were written mainly by @dalecurtis @Djuffin @sandersdan @tguilbert-google @chcunningham. Can any of you say something about how complete this test suite currently is?

[sandersdan]

The test suite does not test every single specified detail; the spec is flexible enough that it is difficult to do so. It was mostly developed in parallel with Chrome's implementation, so there is substantial coverage, but there may also be a few Chrome-isms that slipped in.

[dalecurtis]

I feel it's pretty robust. We tried to add tests for every feature or corner case of the API during development. @padenot indicated it was working pretty well for Mozilla so far. Any feedback you want to add Paul?

That said, there are some gaps in interop between various potentially hardware accelerated features like canvas, hardware decoding, hardware backed capture and how they interact with VideoFrame and the decoders, encoders.

Unfortunately Chromium's WPT system isn't running on systems with GPU backing, so we had to write those tests in a different framework (called GPU tests in Chromium). Basic tests exist in the wpt suite, but different hardware can make all the difference in some cases. I think this is just a limitation of Chromium's WPT setup, so we could port them from the mix of js/python/pixel_difference_harness that they use currently to wpt. @Djuffin who knows more here.

[padenot]

We're still in early implementation, and we're adding tests when we see things missing, and plan to continue adding tests, but generally they've been useful.

[foolip]

In the MDN short survey on APIs & JavaScript, "Web Codecs (AudioDecoder, VideoDecoder, etc.)" was selected by ~21% of survey takers, putting it in the top third of options. (There is some uncertainty as with any survey data.)

[gsnedders]

We said in the last meeting we'd take a closer look at this.

We believe the vast majority of developer interest here is for the video APIs (i.e., VideoDecoder, VideoEncoder, EncodedVideoChunk, VideoFrame, and VideoColorSpace), and as such we'd rather the scope of the focus area be defined as those (i.e., excluding image and audio decoder/encoders). As I understand it, there are negligible performance benefits in the case of audio and images, and it doesn't provide any additional functionality, albeit it does provide them with a cleaner API.

[AshleyScirra]

FWIW, the WebCodecs ImageDecoder provides a way to split animated images (e.g. GIF, APNG) in to separate frames, and the audio encoders allow faster-than-realtime encoding, both of which are not easily possible with other APIs and are important to us.

[dalecurtis]

+1 to @AshleyScirra's comment plus some clarifying points for ImageDecoder:

• AFAIK there's actually no other way on the web platform to extract frames from an animated image short of WASM; by spec drawImage/texImage/createImageBitmap always use the first frame (I was also surprised).
• By the numbers ImageDecoder has more usage than the other APIs (Google Docs being the largest user).

I agree the audio parts are less used at this time, though all RTC related developers have expressed interest. However, until about a month ago the APIs lacked the necessary RTC knobs:

• Add frameDuration attribute to OpusEncoderConfig w3c/webcodecs#551
• Add OpusEncoderConfig parameters w3c/webcodecs#594

[youennf]

I do not dispute the usefulness of all WebCodec features.
AIUI, WebCodecs video support is enabling applications that are simply not possible otherwise(say HD support where WASM is not an option). It does not seem to be the case for the other parts of WebCodecs.
Hence the proposal to make video higher priority/make short term efforts on testing/ensuring WebCodecs video conformance.
The other WebCodec parts could then follow.

[nairnandu]

Thank you for proposing WebCodecs API for inclusion in Interop 2023.

We are pleased to let you know that this proposal was accepted as part of the Web Codecs (video) focus area. You can follow the progress of this Focus Area on the Interop 2023 dashboard.

For an overview of our process, see the proposal selection summary. Thank you for contributing to Interop 2023!

Posted on behalf of the Interop team.

[dalecurtis]

Thanks! I worked on this over at https://bugs.chromium.org/p/chromium/issues/detail?id=1412203 -- when 112 releases to dev channel we should be at 100% on the Chromium side.