Proposal: videoWorklet [worklets-1]

[guest271314]

Given a single or list of URL's videoWorklet does not render media but rather, similar to Web Audio OfflineAudioContext startRendering() https://webaudio.github.io/web-audio-api/#OfflineAudioContext

does not render to the audio hardware, but instead renders as quickly as possible

essentially the opposite of paintWorklet, provides the images to main thread or another context (Worker, SharedWorker, ServiceWorker, window, etc.) as an array of ImageBitmap OR MediaStreams.

Consider the capabilities and functionality of HTML <video> element - without having to load an entire HTML document to access the underlying Media Decoder, Media Encoder, Web Media Player C++ code implemented by browsers (at least Firefox and Chromium) in a Worklet.

Similar concepts and art

• https://github.com/w3c/mediacapture-worker

• https://domenic.github.io/streaming-mediastreams/

• Proposal: Implement OfflineMediaContext whatwg/html#2824

• Can we use HTMLMediaElement or equivalent object without a document? whatwg/html#2981

for the ability to create media such as https://creativecommons.org/about/videos/a-shared-culture and https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm (a video or image collage) using only the browser - without third-party code.

A rough-draft proof of concept that does not meet the specific requirement though provides a tentative model of how controlling a <video> element in a Worklet could resemble, using JavaScript (Ecmascript specification) Module exported and imported https://plnkr.co/edit/5bvp9xv0ciMYfVzG?p=preview.

The videoWorklet can be stateless though still have the ability to send and receive messages.

Ideally, since we should not have to wait for the video to be played at a frame rate for the user to view, the images encoded in the media file (any container or codec the browser is already capable of parsing) should be able to be extracted faster than real-time (at least faster than playback and capturing a MediaStream from a DOM element and MediaRecorder).

[tabatkins]

Ok, so it sounds like you're trying to solve a more general problem, of having the ability to manipulate video streams without needing to instantiate them in a video element.

In that case, it seems like you've already got this well-handled by the existing issues on HTML, and/or the other API proposals you listed. This isn't a Houdini issue; it's not about CSS at all.

(This is why I said that Worklets are not a feature in and of themselves; they're a tool used by other specs that need a specific kind of separate-from-the-main-document JS context.)