Implementing offscreen rendering into External Textures

[ronhab]

Current offscreen rendering approach requires to choose between two non-optimal options:

1. Disabling hardware acceleration - so grabbing frames from memory is fast (since they are being generated in system RAM) but the offscreen rendering itself is slower (since it is done by CPU) and some features are disabled (i.e. WebGL and 3D CSS).
2. Enabling hardware acceleration - grabbing frames from GPU memory is slow, but the offscreen rendering is fast (done by GPU) and all features available.

Would it be possible to implement a 3rd approach in Electron, which will allow to use hardware acceleration for rendering the frames, but instead of returning them as memory buffers (thus requiring a copy from GPU to system RAM) return a handle for the GPU texture (or render directly to a texture supplied externally by the user)?

This is similar to a recent change done in CEF:
see here for the PR, and here for the long discussion.

Thanks!

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[codebytere]

Given that that this isn't currently on our roadmap, and in the interest of our issue tracker more closely reflecting our workstreams, i'm going to go ahead and label this a wontfix. However, should someone determine a workable solution we would be potentially open to accepting a PR.

[luan007]

Allowing OSR with zero-copy texture sharing would open up at least two possibilities:

1. To be used in Audio Visual industries & broadcasting industries: i.e sharing texture / mixing textures with external capturers at 60fps & higher, plays nice with tech like syphon / spout / NDI.. Allowing electron to be 'projection mapped' onto buildings.. or to be used in-par with tools like unreal, to provide better UI / UX for live compositions..
   Similar CEF use cases can be seen in VVVV & so on
   Devs might even use this as a realtime graphics tool for after effect / blender to consume

https://github.com/daktronics/cef-mixer
http://www.thegrego.com/2016/05/16/webgl-in-cinder/

2. To be used to implement complex 3D apps & professional VR applications, feeding rendered texture back as webgl texture for another window to consume. This opens up DOM -> VR overlay potential (discussed a lot in VR field & no standing standard to solve this for now, developers have to either use DOM as plain-overlays or draw everything in canvas2d & use as textures, introducing immense complexity for UI heavy apps like BIM & so on)

@ronhab Is there any chance we can start working on this?
@codebytere or may I ask if it is possible to provide a starting point for us to somehow start trying?

For realistic usage, feeding back into other-window within same electron app should be straight forward,
i.e exchanging handles with ipc

In main process

let hndl = offscreenBrowser.getTextureHandle();  //hndl works like nativeImage

In other render process

//hndl is sent via ipc
let texture = channel.getTextureFromHandle(hndl);
texture.asImg(); 

ref: https://magpcss.org/ceforum/viewtopic.php?f=6&t=14026

[benoitlahoz]

Any news on this one?

[reitowo]

It is hard, because all electron views are rendered by CPU (like popups) when offscreen mode is on.
However if we are ok with render without popup and proxyviews, it is possible to copy the texture directly.

[nrkramer]

If anyone has any interest in opening a PR to address this, the rendering library used to manage the GPU buffer is Skia, and a good place to start would be here:

electron/shell/browser/osr/osr_render_widget_host_view.cc

Line 683 in 6697042

bitmap.readPixels(backing_->pixmap());

That's the line of code where the copy is done (hi @brenca). Perhaps a good design would be to add V8 native language bindings to toggle a boolean disabling that copy.

Additionally, V8 API should be added that can help copy out the buffer's GPU info (get it from the GrBackendRenderTargetDesc ideally, I'm not 100% sure if using SkBitmap::getPixels() returns a GPU pointer address). This info is backend-sensitive (as documented, but I think we can assume OpenGL as used by Electron for now). Some extra information, such as the buffer width, height, texture format, etc might need to be copied out. A hackier workaround would be to just copy the pointer address of whatever gl struct contains that info, and document the struct type so that users can access it with OpenGL APIs.

Once the GPU buffer info is out, that can be passed to whatever rendering code via a NAPI/v8 bridge (which must be implemented by the user). Assuming the user uses something like D3D or Vulkan, some compatibility glue could be used to access the external OpenGL buffer, resulting in 0-copy buffer access to the rendered frames.

I'd do this myself, but my company won't let me. I hope someone else can. This would really enable a lot of great applications that use a 3D-engine based frontend, but need the excellent framework the Electron folks here have made for authoring beautiful GUIs.

---

@codebytere Given that I've presented a solution here, would it be possible to at least have an Electron dev validate this would work/provide additional guidance to anyone who endeavors to build it? I spitballed this after 30min of browsing a bit of code, but I'm not familiar with this codebase at all.

[patacoding]

any news here?

[reitowo]

any news here?

Yes, indeed. In fact I just finished the feature at CEF side few hours ago, see:

https://chromium-review.googlesource.com/c/chromium/src/+/5265077

(Probably OBS, CEF, Electron can unite to pressure Google merging this LOL)

https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=19404&start=20

https://bitbucket.org/chromiumembedded/cef/pull-requests/285/reimplement-shared-texture-support-for-viz

Last time I check the electron is similar to CEF (Both use FrameSinkVideoCapturer), so I think it is definitely portable to electron. I'm going to create a PR if possible.

[reitowo]

Good news: All the patches are merged in M124(6367), so implementing it on electron should be easy.

[reitowo]

Started working on this. See #41972