Is it possible to do our codec or media processing based on WASM

[fideltian]

Hello.
we have a requirement that we need do ourselves encode/decoder/packetization ourselves on WASM. And then transport the data via webtransport or RtcQuic. In the begining, we might need do some media process, besides the encoder/decoder.

What we need is :
1) read the raw data of captured media data, such as YUV video data, or PCM audio data,etc. Then the raw data could be passed to wasm/js module.
2) It will be better the capture could be worked on worker.
3) The data passing is efficient without memory copy.
4) hardware accelerate encoder/decoder could be used by wasm/ js module, which is a Plus.

Could this spec satisfy our requirement?

[guest271314]

1. read the raw data of captured media data, such as YUV video data, or PCM audio data,etc. Then the raw data could be passed to wasm/js module.

It is not clear why it is expected that the "raw data" of a video is "YUV" (see https://stackoverflow.com/a/39048445; https://github.com/brion/yuv-canvas).

The audio data can be expressed as an AudioBuffer using decodeAudioData()

3. If can decode is true, attempt to decode the encoded audioData into linear PCM. In case of failure, set can decode to false.

or a Float32Array using AudioWorkletNode, see https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js (see WebAudio/web-audio-api#2011 relevant to options at the code).

2. It will be better the capture could be worked on worker.

What does "capture" mean in this context?

3. The data passing is efficient without memory copy.

The term "efficient" is not clearly defined. "efficient" compared to precisely what approaches that are not "efficient"?

4. hardware accelerate encoder/decoder could be used by wasm/ js module, which is a Plus.

Not sure what that means. Encoding and decoding any unknown audio and/or video codec by "wasm/js"?

[fideltian]

I did the wrong action of closing the issue.

1. we need to do the vidoe processing ourselved based on WASM. So we might need the RGBA or YUV data of video from WEBRTC capture. We could get the raw data of audio through webaudio API.

2. Capture here means "WebRTC Capture audio/video". It means we would like to get the raw data from worker scope.

3. some CPU has some hardware accelerating encoder/decoder. It will be better to expose the capability to applicaiton layer.

[guest271314]

You can currently use ImageCapture.grabFrame() to get ImageBitmap instances of the captured video frames then process the frames. That procedure is possible right now, without waiting for WebCodecs to specify anything.

[fideltian]

The API returns ImageBitmap , which is not the raw data. It must be drawed to canvas and then get the imageData from Canvas API.

[murillo128]

You can read the image data via the VideoTrackReader.readable and implement your encoder as a TransformStream

[chcunningham]

Old bug! Fidel and Chrome folks have discussed this directly, but for other folks reading along:

What we need is :

1. read the raw data of captured media data, such as YUV video data, or PCM audio data,etc. Then the raw data could be passed to wasm/js module.

Yes, originally offered via VideoTrackReader / AudioTrackReader, but this is now deprecated in favor of (as of #131) using MediaStreamTrackProcessor. Demo here.

2. It will be better the capture could be worked on worker.

Using MediaStreamTrackProcessor, you will have the option to transfer the readable stream to a worker.

3. The data passing is efficient without memory copy.

Working with WASM unfortunately imposes some copies, as described in WICG/reducing-memory-copies#1

4. hardware accelerate encoder/decoder could be used by wasm/ js module, which is a Plus.

At least in Chrome, we will prefer acceleration by default where available. PR #118 will add the ability to strictly require that hardware acceleration be used (which will fail to create the codec when acceleration is not available).

Please re-open if more to discuss.