BabylonNative does not provide a mechanism to get access to its frame buffer

[hendryems]

What's the mechanism for an React Native app to be able to get the frame/(back) buffer for the purposes of streaming out the contents of EngineView in real time?

[hendryems]

Is this dependent on #363?

[ryantrem]

As far as I can tell, WebXR does not currently support getting access to the raw camera frame or the fully composited frame (immersive-web/proposals#36). This likely means we need to introduce a custom API that initially will only work with Babylon Native.

After some discussion, a good first step for this would probably be to expose a JavaScript API (implemented by Babylon Native) that simply provides an ArrayBuffer (rbga raw bytes) and perhaps a stride (to be able to deduce width/height of the image). This effectively means copying the content of the frame buffer from gpu memory to main memory, which is fine for the scenarios we have so far.

By having a JavaScript ArrayBuffer, it is also easy to pass this object to native code (e.g. another Babylon Native plugin, a React Native module, etc.) without any API dependency in the native/C++ layer. This significantly simplifies the programming model.

We already have a way of getting the raw rgba data that is used in the validation tests. We should try to understand whether this is the right way to get the rgba data for a public API that can support things like 30fps video encoding.

We also will need to change the way XR works for ARKit and ARCore to make this work in XR mode. Today we have Babylon render to a texture that is cleared to transparent each frame and then composite that texture with the camera texture outside the context of Babylon Native's NativeEngine (and bgfx). This means those layers don't have access to the final composited image. Instead, we should "render" the part of the display (control) cropped camera texture into the Babylon texture, and then let Babylon render on top of that texture. Then, the final composited image will be accessible through the NativeEngine, and therefore we can expose an API that can get rendered pixels regardless of whether we are in XR mode.

[dylanleatham]

Tech Assist has a need for this feature to support our calling and video recording scenarios. For our asset capture feature, we want to access the video frame buffer (with mixed reality annotations) and combine that with captured audio data to produce a video file of the scenario. In a call, we will be streaming the video buffer through IC3 to other call participants. We do not need to handle streaming the video buffer over a call AND recording the frame, at least for our P0