rfc: Import Shared Texture

[reitowo]

This feature provides a way to import a native shared texture handle into Electron, specifically in the form of VideoFrame, which by nature supports several Web rendering systems including WebGPU, WebGL. This enables developers to integrate arbitrary native rendered content with their web applications.

[samuelmaddock]

Are there any use cases for keeping the imported texture in the main process after initiating the transfer to the renderer?

I wonder if the API could be simplified where imported would become unusable in the main process if sent over IPC. This would be similar to transferrable objects. It'd also be valid to limit the imported shared texture to only be usable with MessagePort which accepts transferrable objects.

[reitowo]

1. It may be transferred to multiple renderer processes. For example, display this texture in multiple pages. The word "transfer" here does NOT mean handling over ownership like "transferrable objects" means. Considering this, we can not use MessagePort or anything that transfer ownership.
2. We must wait the receiver to call finishTransferSharedTexture, before call release at sender side. More precisely, we have to hold reference until the GPU actually executed the import task for receiver. That's when it adds reference count to the underlying SharedImage & Mailbox. If we call release here right after startTransferSharedTexture, we submit a release task to GPU, and this task may be executed before the import task, which makes the reference count returning 0, and the SharedImage & Mailbox get finally destroyed, causing the import task fail.
3. The IPC to notify main process that is OK to release is unavoidable.
   • Although we can pass a SyncToken back to sender (either manually or automatically via some mojo callback), to mark the import task at receiver side as a dependency of the release task at sender side, which prevents the reference count returning to 0. We still need to hold the reference at sender side, until this SyncToken is passed back to main process. Which no matter how needs a IPC event to notify.
   • Chromium internally share SharedImage between process use similar approach, but the key difference is, they establish this mojo callback to pass SyncToken back, at the same time they pass the ExportedSharedImage to receiver process, which Electron JS API cannot achieve easily. And, they first establish a IPC channel with receiver at another process, and pass this ExportedSharedImage using that channel, which can ensure the remote side called import and get the SyncToken back syncly. In Electron IPC, we don't have such channel and can't do this syncly. One example is: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/platform/graphics/accelerated_static_bitmap_image.cc;drc=02c21516ef901e09640e417bde686ba469d7438a;l=83 the external_callback is for passing SyncToken back, while https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/public/common/messaging/accelerated_image_info.h;drc=e76cd1dd569db9198eb674102f00a718a752487d;l=23 itself is passed by other IPC channel.
   • Since it is unavoidable, implementing automatic SyncToken callback doesn't have that much meaning, so I choose to provide a callback in release function to notify the SyncToken is executed and it is safe to release dependent things. (e.g. notify sender process to call release).

[samuelmaddock]

Could handling the release of these textures be handled internally? I can think of two different ways to implement this:

1. Collect capturedTextures within lib/ and use an internal IPC to release these.
2. When release() is called from the renderer, send an internal Mojo IPC to release the native data structures.

[reitowo]

I also considered this, and I found if we want to let user transfer same texture to (multiple) different processes, we eventually creates a cross process reference counting map, which is fun, because in fact, the same thing SharedImage is doing. It use Mailbox as a key to reference counting and holds the SharedImage data. https://source.chromium.org/chromium/chromium/src/+/main:gpu/command_buffer/service/shared_image/shared_image_manager.h;drc=e76cd1dd569db9198eb674102f00a718a752487d;l=32

But, SharedImageManager has a drawback that I mentioned in the above conversation, the reference counting add/sub is not executed until GPU process run that command buffer. If we're able to rebuild the wheel, we true may creates a "sync" version of reference counting, and yes, we don't have to let user pass a callback to release function and manually notify main process to release. The only problem is: we have to syncly add the reference count at the time "startTransferSharedTexture" and hope the receiver will catch that thing. The ownership is not taken until receiver receive the object. If somehow the texture is lost, and release never get called, the texture is never released

The procedure will be changed to:

1. User import shared texture at main process, we create a record and add reference
2. User start transfer, we add reference syncly.
3. User release it at main process as no longer needed. The texture is not released as it still have a reference count.
4. User use finish transfer at renderer process to get it back, we do nothing on counter (already added at sender side)
5. User finished accessing it at renderer process, call release, and we decrease the reference.
6. The counter is now 0 and we eventually free the SharedImage.

The problem is:

1. If somehow a started transfer texture never being received by user, it never get released, the SharedImage is memory leaked.
2. We also can't wait to add reference count at finish transfer side, which requires main process to hold until this happens, and renderer process notify main to release, which makes no difference than current design.
3. Performance, don't know how long it takes to update the counter. If there's a OS level counter to use?
4. Repeated design of the counter as Chromium internally does that, although deferred executed (which is not ideal.)
5. We maybe have to listen to multiple GC callbacks to detect if user leaked the texture, or automatically release for user. As this is a cross process holder, and we are dealing with multiple GC environments.

[reitowo]

Frankly speaking, a native implementation of cross process reference counting is not simple. Let's say it is named CrossProcessCounterService, in order to do that natively, we have to create a host service in main process, when creating subprocesses we have to pass mojo remotes to new process and creates a ClientInterface for this in order to communicate with host. All of these implementation is for provide a cross process counter, seems a little bit complicated.