Skip to content

Add multiview support for XRProjectionLayer - #11828

Merged
RaananW merged 8 commits into
BabylonJS:masterfrom
rgerd:xr-proj-multiview
Jan 26, 2022
Merged

RaananW merged 8 commits into
BabylonJS:masterfrom
rgerd:xr-proj-multiview

Conversation

@rgerd

@rgerd rgerd commented Jan 24, 2022 •

Copy link
Copy Markdown
Contributor

This change hooks up the XRProjectionLayer to our existing xr multiview rendering implementation. The "texture-array" texture type in XRProjectionLayerInit is designed to be used with the multiview WebGL extension, and so we use multiview rendering whenever "texture-array" is specified as the texture type.

Before:
scene-render-no-mv

After:
scene-render-mv

Taking performance profiles on an Oculus quest with a playground rendering 400 default-fidelity spheres (https://playground.babylonjs.com/#F41V6N#705), I got the following per-frame times for multiview turned off and on:

Multiview off

  • scene.render: 59-62ms
  • GPU time: 5ms
  • renderFunction (the entire frame): 61-72ms

Multiview on

  • scene.render: 30-32ms
  • GPU time: 3ms
  • renderFunction: 32-40ms

Exported profiles from Chrome DevTools:
xr-multiview-profiles.zip

So we got some significant performance wins on both the CPU and GPU, as would be expected. With scene.render taking up around 95% of the entire time for the frame, cutting the time for that function down in half just about doubled the overall FPS.

Even with one sphere, I'm still seeing around a 10% decrease in frame time (40% in scene.render) and about a 20% decrease in GPU time.

fixes #10767

@rgerd
rgerd requested a review from RaananW January 24, 2022 08:11
Comment thread src/XR/features/WebXRLayers.ts Outdated

@RaananW RaananW left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this is so great :-)

What would be the way to use this?
I saw you mentioned a flag. I think semantically putting the flag in the XR layers feature might be a little confusing. Though it is directly related to the feature.

We want to make it as simple as possible to use the feature, while allowing as much flexibility as possible.

Comment thread src/XR/webXRCamera.ts Outdated
Comment thread src/XR/webXRRenderTargetTextureProvider.ts
Comment thread src/scene.ts
@rgerd

rgerd commented Jan 25, 2022

Copy link
Copy Markdown
Contributor Author

So the main idea is that you'd specify multiview for the layer you're creating, and that layer would render using multiview:

layersFeature.createProjectionLayer(params, true);

(It seems we might also be able to use multiview for quad layers down the road)
Multiview only works if the textureType is 'texture-array'

The one thing to note is that we don't currently have a way of doing non-multiview if the textureType is 'texture-array' because we don't have a way to save the sub image index in a render target texture. We can bind the RTT's framebuffer to a sub image index for gl.TEXTURE_2D_ARRAY, but there's currently no way to get the correct image index for the view when we want to render a specific eye.

I think for now, to avoid a path where the user creates a projection layer with 'texture-array' without multiview and sees broken rendering, it makes sense to me to force multiview whenever the textureType is 'texture-array', and then do another PR to create maybe a RenderTargetTexture subclass that is explicitly bound to the sub image index corresponding to the eye being rendered with that RTT. If multiview isn't supported or the user doesn't want to use multiview, then the user can still use textureType 'texture', or do nothing since the layers feature doesn't use multiview by default.

@RaananW

RaananW commented Jan 25, 2022

Copy link
Copy Markdown
Member

I think for now, to avoid a path where the user creates a projection layer with 'texture-array' without multiview and sees broken rendering, it makes sense to me to force multiview whenever the textureType is 'texture-array', and then do another PR to create maybe a RenderTargetTexture subclass that is explicitly bound to the sub image index corresponding to the eye being rendered with that RTT. If multiview isn't supported or the user doesn't want to use multiview, then the user can still use textureType 'texture', or do nothing since the layers feature doesn't use multiview by default.

Let's create an issue and follow that, as this can be very confusing for a developer trying to use a texture-array.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[XR] Enable MultiView mode

2 participants