WebCodecs Position

[sandersdan]

WebCodecs deals with a few kinds of buffers: encoded packets, video frames, and audio buffers. Video frames are at least an order of magnitude larger than packets and audio buffers, so I focus on them below.

Video frames are typically backed by GPU memory, using the native graphics buffer primitive (eg. DMAbuf, IOSurface, etc). These buffers can be mapped for reading but usually must be locked during access. Video frames may also be stored in CPU memory.

We assume that video frames are immutable due to lifetime and security restrictions imposed by the underlying codec implementations.

In each case I list the most obvious solution as an example, but there are known technical problems with each of them.

WebCodecs <-> JS

We would like JS to be able to read planar image data from video frames, and to be able to construct new video frames from planar data. We can't use ArrayBuffer for this as ArrayBuffers are always mutable.

The current WebCodecs proposal provides a readInto() method to copy data from a video frame into a destination ArrayBuffer, and copies from a source ArrayBuffer to create new video frames.

The most obvious solution here is read-only ArrayBuffers.

WebCodecs <-> WASM

The WASM situation is the same as JS, except that ArrayBuffers are not available (other than the WASM main memory).

The most obvious solution here is a memory mapping facility for WASM.

WebCodecs <-> <canvas>, WebGL

The web-native interop format for <canvas> and WebGL is ImageBitmap, and we currently believe that we can offer zero-copy ImageBitmap in most cases.

Using the "imagebitmap" context, it should be possible to render an ImageBitmap also with no additional copies (via transferFromImageBitmap()). In WebGL however there is no way to make use of an ImageBitmap without copying it (via TexImage2D()).

It is also possible to upload and read back data using pixel pack buffers, which may be efficient enough in the case of video frames that are backed by CPU memory.

The most obvious solution for WebGL is a way to bind video frames to textures, as in the WEBGL_video_texture extension.

WebCodecs <-> WebGPU

WebGPU has a concept of buffers that matches platform graphics buffers. We are hopeful that it will be possible to interoperate in that way, but there are significant open questions and a few problematic corner-cases.

Some discussion here: gpuweb/gpuweb#625, gpuweb/gpuweb#700, gpuweb/gpuweb#1154.

A potential solution here could be an enhanced ImageBitmap-like object that WebCodecs can export and WebGPU can import.

[littledan]

We're considering read-only ArrayBuffers as part of https://github.com/tc39/proposal-readonly-collections . The discussion in that proposal has focused on maps and sets, so more context for use cases for ArrayBuffers is appreciated. cc @phoddie

[phoddie]

Yes, thanks for providing another example where read-only ArrayBuffers would be valuable. The read-only collections proposal does include that, though the Stage 1 proposal text mentions that only in the closing note.

There are a couple more ways that read-only ArrayBuffers might find their way into the language:

• A more general approach to read-only objects is be possible by extending the Record and Tuples proposal. This is likely more complex to specify but has more benefits.
• A simpler solution than either of these is to provide read-only ArrayBuffers. That seems viable given recent work in TC39 on growable ArrayBuffers.

[phoddie]

I'm not sure. ;)

I think JavaScript deserves consistent handling of read-only objects. That's a discussion we are having. I won't open a new front on that discussion here.

[syg]

Read-only ArrayBuffers have come up in discussion before:

• List of issues to discuss with TC39, WASM, JS engine teams gpuweb/gpuweb#747 (comment)
• read-only ArrayBuffer is strange WebBluetoothCG/web-bluetooth#300

The particular past concerns are about performance, which the ResizableArrayBuffer proposal also contends with. Specifically, Ken Russell's concerns come from his expertise and past experience with handling permutations of ArrayBuffer "modes" like read-only in Java, where the polymorphism caused deopt and significant performance hits in JVM.

Polymorphism in JS is handled differently via per-use-site type feedback, so it might be less of an issue for us. However, the core issue exists: a single ta[idx] access now might be polymorphic not just because of the TypedArray type but also because of the underlying buffer type.

The hypothesis for resizable arrays is that most applications would not be mixing resizable and fixed arrays at the same use sites. What are folks' intuitions for mixing TAs backed by read-only buffers vs mutable buffers?

[sandersdan]

On a per-callsite basis, I expect that WebCodecs users would likely be running pipelines that process VideoFrames using the same code path thousands of times, and would not be mixing in other kinds of buffers in the same pipeline. There is a risk that some part of the same app would be simultaneously using RW buffers, in which case inlining of any utility helpers could be a factor.

[kenrussell]

As mentioned above in gpuweb/gpuweb#747 , some background on read-only ArrayBuffers and Typed Arrays was posted on the internal WebGPU mailing list:

https://lists.w3.org/Archives/Member/internal-gpu/2020May/0001.html

In one reply to that email (which I'm not sure was archived), Filip Pizlo from WebKit's JSC team confirmed that there would be a significant performance hit on all typed array accesses if read-only ArrayBuffers or Typed Array views were added, and used anywhere in the same program:

https://lists.w3.org/Archives/Member/internal-gpu/2020May/0003.html

If desired, these discussions can be copied to a more public place - please comment here if so.

Two more complexities involved in adding either read-only ArrayBuffers or Typed Arrays:

• WebAssembly's memory is represented as an ArrayBuffer. There's no concept of referencing a sub-portion of an ArrayBuffer as another one - that's what typed array views do. Therefore there would be no way of treating a portion of the Wasm heap as a read-only ArrayBuffer, or forbidding Wasm code from writing to its heap. Interaction with Wasm is a key use case.

• It's always possible to traverse from the Typed Array to the ArrayBuffer which it references. If writeable ArrayBuffers could have read-only Typed Array views created against them, then it's not clear what would prevent a writeable Typed Array from being created against the same ArrayBuffer.

[syg]

Thanks for the extra info! I have trouble accessing lists.w3.org even after repeatedly resetting my password, so a copy/paste would be appreciated. I have seen those emails before though, if they are the ones I think they are:

In one reply to that email (which I'm not sure was archived), Filip Pizlo from WebKit's JSC team confirmed that there would be a significant performance hit on all typed array accesses if read-only ArrayBuffers or Typed Array views were added, and used anywhere in the same program:

I'd be interested to learn why Fil thinks that is -- is it the callsite polymorphism problem or something else? He talks about assuming that a program will definitely use both buffer types.

WebAssembly's memory is represented as an ArrayBuffer. There's no concept of referencing a sub-portion of an ArrayBuffer as another one - [...]

Is this mprotect use case for sub-portions the actual use case for WebCodecs? If so, then I agree read-only ArrayBuffers aren't a solution. Because wasm exposes the entire linear memory, that use case requires an mprotect-like utility directly, I imagine. (I thought the use case here was creating entire read-only buffers from the start.)

It's always possible to traverse from the Typed Array to the ArrayBuffer which it references. If writeable ArrayBuffers [...]

I completely agree that read-only TypedArray views aren't a very useful thing for the reason you give. I've been assuming the ask here is for read-only ArrayBuffers.

[littledan]

It sounds like Filip is agreeing with Ken that, if we had code which used read-only and read-write ArrayBuffers in a mixed way, it would lead to slower performance. (I'm not sure if whether it's appropriate for me to publish the email; just write me a message if you want a link to a secret gist with the contents.)

Note that, in TC39, we're considering a different source of having multiple types of ArrayBuffers, in @syg 's ResizableArrayBuffer proposal. This proposal notes that it could lead to polymorphism.

I could imagine two different semantics for read-only ArrayBuffers:

• read-only view of an ArrayBuffer that might change out from underneath (perhaps from an operation on an ArrayBuffer to return another ArrayBuffer which cannot be written to, but which may alias another one)
• a snapshot of an ArrayBuffer that won't change at all (perhaps the result of a special freeze operation on a "freezable ArrayBuffer" which is initially mutable).

The read-only collections proposals provides both of these semantics, through the snapshot vs readOnlyView operations. I'm curious whether one or both of these is higher priority for the Web.

[lukewagner]

Hi! If I could ask for a bit more background information: my default assumption would be that the representation of video frames in host memory would vary based on OS, hardware, video format, etc. If that is the case, would WebCodec (1) expose this memory representation variance directly to JS/wasm content, or (2) perform a normalizing conversion to a standard representation (e.g., like ImageData.data)? Then a follow-up question would be:

• if (1): would the spec enumerate a finite set of representations covering most situations (avoiding the need for a conversion from the underlying native resource in the common case)?
• if (2): is there a way that this conversion could be delayed so that it can fused with the operation that copies the data into wasm's linear memory?

[sandersdan]

WebCodecs is mostly like (1), if an app doesn't understand the native format then it may need to use a less efficient fallback (generally conversion to an ImageBitmap). This so far only applies to memory layout, the copyInto() API we are currently using abstracts some details.

We expect that removing the remaining copy operation will require multiple representations, but the fewer the better of course.

WebCodecs is probably somewhat unique in this regard, our users (so far) have substantial development teams that can implement a variety of optimizations in their apps, and they are willing to commit those resources to improve performance.

[lukewagner]

Thanks!

So, just to have all the options on the table, there is a long-standing wasm future extension we've discussed in the CG that would effectively let wasm directly use typed array views or array buffers, providing an alternative to the mmap-into-linear memory approach. The idea is to add a sliceref (strawperson name) value type along with instructions to perform loads or stores from this sliceref. The key limitation of this approach (which is why we haven't seriously pursued it yet) is that, since the sliceref is distinct from wasm's default linear memory, you can't use normal C/C++ pointers to access it (since they are all implicitly relative to the default linear memory). Rather, clang would need to be extended with some sort of compiler intrinsics for (1) passing around sliceref values (so, some sort of magic C/C++ type that clang compiled down to a sliceref), and (2) performing sliceref loads/stores (so, e.g., builtin functions like:int32_t __builtin_load_i32(sliceref base, uint32_t offset)). You could hide these behind a more-idiomatic-looking C++ smart-pointer type, but it would still mean carefully (re)writing the code that needed to access this video frame memory. This is a big hurdle but, from what you're saying, maybe the relevant dev teams would be up for it?

[sandersdan]

Hmm, I'm unsure. I expect our users would be willing to make such changes at the layer where they interface with WebCodecs, but they are also likely to have existing media processing code (eg. bundled ffmpeg filters) that expects regular pointers. Probably only some of our users are willing to invest in changes that large.