API for containers?

[pthatcherg]

It comes up as a common question: can we have an API for media containers? It's something that can be done JS/wasm and is arguably orthogonal to WebCodecs. But for some formats that you might consider video (GIF, (M)JPEG), the line is blurry between container and codec.

This is a tracking issue for a conversation around this topic. My current opinion is to leave it out of WebCodecs until it's more mature and then perhaps readdress it later.

[steveanton]

The concern I have with containers is that it exposes a lot of API and specification surface but does not unlock anything new that can't already be done (pretty efficiently even) today.

Following the principles of the Extensible Web Manifesto (https://extensiblewebmanifesto.org/), we should focus on delivering low-level codecs first. Only after that's done (or in parallel by a different group) consider tackling containers.

[pthatcherg]

I agree completely. That's a great way to put it.

[guest271314]

Key use-cases

• Non-realtime encoding/decoding/transcoding, such as for local file editing
• Decoded and encoding images
• Reencoding multiple input media streams in order to merge many encoded media streams into one encoded media stream.

and potentially

• Live stream uploading

each could be considered an API that at least in part performed some form of code to edit a file. A file could be considered a "container" when technically compared to

• Extremely low latency live streaming (<3s delay)
• Cloud gaming
• Advanced Real-time Communications:
  -- e2e encryption
  -- control over buffer behavior
  -- spatial and temporal scalability

and potentially

• Live stream uploading

where any and all of the above is encompassed within the use case of recording media into a container whether that "container" be an array of images with an index element for "metadata", i.e., width, height, frame duration, or other adjustments made "mid-stream" or post-production ("codec" or instruction), to a .json (or, if preferred Matroska or WebM) "container", for download of both the entire procedure output by WebCodecs and specific time slices into a single "container" (file structure).

Since the topic is at hand and the maintainers of this repository spans a wide range of topics it might be helpful to create a glossary to point to exactly what you (this repository) mean the definition of the term that you are using.

Non-goal

• Direct APIs for media containers (muxers/demuxers)

indicates the technical proximity of "codecs" to "containers".

Whether or not WebCodecs includes the reading, writing, editing, etc. of both "codecs" and "containers" that internal decision will not prevent nor preclude the fact of the actual use-case for a single API which is capable of both codec and container creation, extension, modification, etc., to avoid the necessity of attempting to utilizing what is available in different specifications that were not initially conceived as being interoperable with other APIs, both existing and proposed.

One recent use case for not omitting to include the capability to perform the same procedures within the scope of "containers" the same as "codecs" is that the two are symbiotic and when a single API designed and maintained with that consideration at the forefront and throughout has the potential to solve more than one existing issue where separate portions of code in the same domain, for example, media, could have very different output due to different authors' intent at the time they wrote and merged the code: time passes, new technologies that the issue resolves are now an issue because the implementers may or may not be in accord with the various branches of "media". A single API from creation to editing to production of media streams and files is the explicit use case.

[guest271314]

Example use case: write audio to a WebM file https://plnkr.co/edit/Inb676?p=preview. Ideally, from a front-end perspective, this single API should be able to encode VP8 and Opus to a file, or if Opus is missing from an existing file, write the audio to the file.

[guest271314]

@steveanton

(or in parallel by a different group)

What is necessary to start such a group? Post the proposal at https://discourse.wicg.io? (Note, am not a member of W3C).

[guest271314]

@pthatcherg

It's something that can be done JS/wasm

While there does exist code which can write input images and output a WebM file (https://github.com/GoogleChromeLabs/webm-wasm; https://github.com/thenickdude/webm-writer-js) neither of the repository authors are interested in implementing writing audio to the same output file

thenickdude/webm-writer-js#8 (comment)

I don't have any plans to work on adding audio, sorry, and I'm not sure where best to begin either (it probably depends on what format you can capture the audio in and what environment you expect to run in, Chrome, arbitrary browser, Electron, etc).

GoogleChromeLabs/webm-wasm#12

It seems this project doesn't support encoding audio+video yet, just video-only? It this feasible, or would it be better to just use the more heavyweight ffmpeg.js project for this?

GoogleChromeLabs/webm-wasm#12 (comment)

Yeah, there’s no support for audio and I don’t have any plans to add it. This project was born out of the lackluster capabilities of MediaStreamRecorder.

ffmpeg.js is definitely one choice. But if you already have an encoded audio and video stream, an mkv muxer might do. That would be a lot faster and smaller. Hope this helps!

It seems this project doesn't support encoding audio+video yet, just video-only? It this feasible, or would it be better to just use the more heavyweight ffmpeg.js project for this?

There is an implementation which is capable of writing audio as Opus to a WebM container https://github.com/kbumsik/opus-media-recorder/.

Meaning this repository can reach maturity without a corresponding container writer being available and having proximal maturoty to write the output of WebCodecs to a file.

Thus, simply because "JS/wasm" exists does not mean that implementations exist to meet the requirements described at Key use-cases, particuarly

• Non-realtime encoding/decoding/transcoding, such as for local file editing.

[guest271314]

FWIW https://discourse.wicg.io/t/webmediacontainers-proposal/3928

[padenot]

To write down what was said during TPAC, this might be very important to avoid the proliferation of badly muxed files. Muxing is rather hard to get right.

A possible solution might be a vouched library, as noted above, but there is always the problem of updating it for bug fixes.

[chrisn]

I've been following the general discussion around WebCodecs, and the need for a media container API seems to be recognised, but I thought I'd add my own use case as an example.

I maintain a library waveform-data.js that produces data for waveform visualisation from audio.

This uses Web Audio decodeAudioData() - but has the well-known problems: it runs on the main thread so UI updates stall during decoding, it requires the entire encoded audio to be held in memory, there's no indication of progress so I can't tell how long it will take to complete, and there's no way to cancel the decode.

For this use case, the simplest solution would be to allow decodeAudioData() to run from a worker context, with an extended API to allow progress notifications and cancellation.

WebCodecs also solves these issues, but introduces a new one. Because the library is generic, it will accept any audio format that decodeAudioData supports. So in order to use Web Codecs the library would have to include code to parse all the container formats, or define an API that moves container parsing to users of the library. Both options increase the amount of JavaScript that needs to be delivered, and unnecessarily so because parsing the container is a capability the browser already has. Also, leaving container parsing to library users would make the library much harder for people to use.