WebGLRenderer: Improve offscreen rendering in WebXR.

[CodyJasonBennett]

Fixed #21188, #18846
Related issue: #8146

Description

Binds renderer.setRenderTarget(null) to renderer.setRenderTarget(_layerRenderTarget) in WebXR. This fixes scenes which render offscreen and composite like post-processing, GPGPU (e.g. LUTs), and volumes. This can be worked around by calling renderer.getRenderTarget at the start of each frame and binding to that instead which is effectively what this PR does.

I've also updated fixes from #18846 for the new layers API so renderer.getDrawingBufferSize and renderer.getSize will report correctly in WebXR, resolving #21188 (cc @gkjohnson). Note that renderer.getPixelRatio will always report 1 regardless of native scaling factor (as implemented).

This demo works around the above issues, rendering a grid to a render target and blitting it: https://codesandbox.io/s/mdhg6s

Before, that would be:

const handleResize = () => {
  renderer.setSize(window.innerWidth, window.innerHeight)
  renderTarget.setSize(window.innerWidth, window.innerHeight)
  camera.aspect = window.innerWidth / window.innerHeight
  camera.updateProjectionMatrix()
}
handleResize()
window.addEventListener('resize', handleResize)
renderer.xr.addEventListener('sessionend', handleResize)

// https://github.com/mrdoob/three.js/issues/21188
const resizeWithBaseLayer = () => {
  const layer = renderer.xr.getBaseLayer()
  const width = layer.textureWidth ?? layer.framebufferWidth
  const height = layer.textureHeight ?? layer.framebufferHeight
  renderTarget.setSize(width, height)
}
renderer.xr.addEventListener('sessionstart', resizeWithBaseLayer)

renderer.setAnimationLoop(() => {
  controls.update()

  // In the DOM, gl.bindFramebuffer(gl.FRAMEBUFFER, null) binds
  // to the canvas, but in WebXR we render straight to a layer.
  //
  // Three.js creates an internal render target, so calling
  // renderer.setRenderTarget(null) will not render to screen
  // in WebXR.
  //
  // Layers will also have a different size, so mirror the
  // render target's dimensions which correspond to the layer.
  const _surface = renderer.getRenderTarget()

  renderer.setRenderTarget(renderTarget)
  renderer.render(scene, camera)

  // Disable XR projection for fullscreen effects
  const xrEnabled = renderer.xr.enabled
  renderer.xr.enabled = false
  renderer.setRenderTarget(_surface)
  renderer.render(effect, camera)
  renderer.xr.enabled = xrEnabled
})

Now, that would be:

const size = new THREE.Vector2()

const handleResize = () => {
  renderer.setSize(window.innerWidth, window.innerHeight)
  renderer.getDrawingBufferSize(size)
  renderTarget.setSize(size.width, size.height)
  camera.aspect = window.innerWidth / window.innerHeight
  camera.updateProjectionMatrix()
}
handleResize()
window.addEventListener('resize', handleResize)
renderer.xr.addEventListener('sessionstart', handleResize)
renderer.xr.addEventListener('sessionend', handleResize)

renderer.setAnimationLoop(() => {
  controls.update()

  renderer.setRenderTarget(renderTarget)
  renderer.render(scene, camera)

  // Disable XR projection for fullscreen effects
  const xrEnabled = renderer.xr.enabled
  renderer.xr.enabled = false
  renderer.setRenderTarget(null)
  renderer.render(effect, camera)
  renderer.xr.enabled = xrEnabled
})

---

Regarding examples and EffectComposer, I've since mirrored the above demo -- resizing with the base layer and disabling XR projection for fullscreen effects via FullScreenQuad.

I have a branch based on this one which enables WebXR for all of the postprocessing examples (diff) which you can try at https://rawcdn.githack.com/CodyJasonBennett/three.js/c13f9091360c8eda4c198f164d2c63eb1b3986a0/examples/?q=postprocessing#webgl_postprocessing. Notably, (S)SAO/Background don't work because of CustomBlending and unexpected clearing with WebGLBackground from the XR environment blend mode (fixed since #26428).

Show examples:

Below are taken with an emulator, but this PR ensures it also works on-device.

Example	Preview
webgl_postprocessing
webgl_postprocessing_3dlut
webgl_postprocessing_afterimage
webgl_postprocessing_dof
webgl_postprocessing_fxaa
webgl_postprocessing_glitch
webgl_postprocessing_rgb_halftone
webgl_postprocessing_sobel
webgl_postprocessing_unreal_bloom
webgl_postprocessing_unreal_bloom_selective

---

WRT previous concerns for performance, I haven't opted for adding any external examples for pmndrs/postprocessing since #18846 (comment), but I have tested against this PR's builds with the library against a few R3F examples that use it (e.g. https://codesandbox.io/s/7c35ym). I'd recommend this library since it has very performant effects that I've been able to run on low end mobile hardware, and they're modular enough to mix with three's if you really want to. Notable user-land integrations include 0beqz/realism-effects and N8python/n8ao.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

Filesize dev	Filesize PR	Diff
649.2 kB (160.9 kB)	649.5 kB (161 kB)	+351 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

Filesize dev	Filesize PR	Diff
442.5 kB (107.1 kB)	442.8 kB (107.1 kB)	+351 B

[cabanier]

This looks like a good change but it would be nice to see how it performs on mobile hardware.
Can you also add a vr example that invokes these new codepaths? (or is there one already?)

[CodyJasonBennett]

This PR shouldn't have any performance implications since it doesn't touch any hot paths. Are you concerned with post-postprocessing end-to-end or something more implementation specific with this PR?

I tested the unreal bloom example which you can try at https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html. Not all effects work like the pixel pass, but I don't plan on addressing them here but separately since this PR contains fixes I'd like to not block.

In user-land, I'd recommend pmndrs/postprocessing since it has very performant effects, but I haven't been able to get a stable enough implementation over WebXR in three.js prior.

[cabanier]

This PR shouldn't have any performance implications since it doesn't touch any hot paths. Are you concerned with post-postprocessing end-to-end or something more implementation specific with this PR?

I am worried about introducing flushes because they will impact performance and introduce visual glitches (ie "snow"). If postprocessing could discard depth or even not create a resolve, that would be better.

I tested the unreal bloom example which you can try at https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html. Not all effects work like the pixel pass, but I don't plan on addressing them here but separately since this PR contains fixes I'd like to not block.

Could you create an example based on that?

[CodyJasonBennett]

If you're referring to internal disposal from resizing effects/render targets in EffectComposer, I can split that into a separate PR. I don't want that to block this PR behind further enhancements.

[cabanier]

If you're referring to internal disposal from resizing effects/render targets in EffectComposer, I can split that into a separate PR. I don't want that to block this PR behind further enhancements.

No, just a simple example that uses these new code paths. That way I can make sure that nothing is broken in our PRs for multiview and our work for spacewarp and timewarp layers.

[CodyJasonBennett]

I've added a minimal demo webxr_vr_postprocessing that uses EffectComposer, UnrealBloom, and the new OutputPass. My PR description has a demo that performs a simple blit via workarounds if you want something smaller and without this PR.

[PhilipZhu]

I tried to change parameters from the control panels in https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html, and regular rendering from the browser window and XR rendering doesn't match. This is true for both the chrome emulator extension and Quest2
Regular broswer window:

VR render:

[marcofugaro]

really useful PR! would love to see it merged

[cabanier]

This is not a good change as it destroys foveation and constantly rebinds the buffer. Performance will be bad which will frustrate developers.
Testing on Quest 2, these 4 triangles can barely be rendered. Looking with ovrgpuprofiler, i can see retargeting of the drawing buffer, flushing of the depth buffer and temporary textures (some of which are full resolution 64 bit). Multisampling also isn't working in the example.

Unless this can be addressed, please revert this for now.
Here is the output of a single frame from ovrgpuprofiler of webxr_vr_postprocessing :

Surface 0 | 2880x1584 | color 32bit, depth 24bit, stencil 64bit, MSAA 1, Mode: 0 (Direct) | 1 2880x1584 bins ( 1 rendered) | 0.00 ms | 1 stages : Render : 0.002ms
Surface 1 | 2880x1584 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 2880x1584 bins ( 1 rendered) | 0.07 ms | 2 stages : Render : 0.062ms Blit : 0.002ms
Surface 2 | 1440x792 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 1440x792 bins ( 1 rendered) | 0.42 ms | 2 stages : Render : 0.416ms Blit : 0.001ms
Surface 3 | 1440x792 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 1440x792 bins ( 1 rendered) | 0.34 ms | 2 stages : Render : 0.336ms Blit : 0.002ms
Surface 4 | 1440x792 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 1440x792 bins ( 1 rendered) | 1.23 ms | 4 stages : Render : 0.549ms Blit : 0.002ms Preempt : 0.673ms
Surface 5 | 720 x396 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 720x396 bins ( 1 rendered) | 1.09 ms | 2 stages : Render : 1.079ms Blit : 0.003ms
Surface 6 | 720 x396 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 720x396 bins ( 1 rendered) | 0.15 ms | 2 stages : Render : 0.149ms Blit : 0.002ms
Surface 7 | 360 x198 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 360x198 bins ( 1 rendered) | 0.38 ms | 2 stages : Render : 0.378ms Blit : 0.002ms
Surface 8 | 360 x198 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 360x198 bins ( 1 rendered) | 0.08 ms | 2 stages : Render : 0.079ms Blit : 0.002ms
Surface 9 | 180 x99 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 2 (SwBinning) | 1 192x112 bins ( 2 rendered) | 0.15 ms | 4 stages : Render : 0.132ms StoreColor : 0.004ms Blit : 0.002ms StoreDepthStencil : 0.003ms
Surface 10 | 180 x99 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 2 (SwBinning) | 1 192x112 bins ( 2 rendered) | 0.05 ms | 4 stages : Render : 0.033ms StoreColor : 0.004ms Blit : 0.002ms StoreDepthStencil : 0.003ms
Surface 11 | 90 x50 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 2 (SwBinning) | 1 96 x64 bins ( 2 rendered) | 0.06 ms | 4 stages : Render : 0.05ms StoreColor : 0.002ms Blit : 0.002ms StoreDepthStencil : 0.002ms
Surface 12 | 90 x50 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 2 (SwBinning) | 1 96 x64 bins ( 2 rendered) | 0.03 ms | 4 stages : Render : 0.017ms StoreColor : 0.003ms Blit : 0.002ms StoreDepthStencil : 0.003ms
Surface 13 | 1440x792 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 1440x792 bins ( 1 rendered) | 0.42 ms | 2 stages : Render : 0.411ms Blit : 0.002ms
Surface 14 | 2880x1584 | color 64bit, depth 24bit, stencil 0 bit, MSAA 1, Mode: 0 (Direct) | 1 2880x1584 bins ( 1 rendered) | 1.82 ms | 3 stages : Render : 1.112ms Preempt : 0.708ms
Surface 15 | 2880x1584 | color 32bit, depth 24bit, stencil 64bit, MSAA 1, Mode: 2 (SwBinning) | 36 480x288 bins ( 30 rendered) | 2.12 ms | 88 stages : Render : 0.998ms StoreColor : 0.472ms Blit : 0.004ms StoreDepthStencil : 0.482ms
Surface 16 | 2880x1584 | color 32bit, depth 24bit, stencil 64bit, MSAA 1, Mode: 0 (Direct) | 1 2880x1584 bins ( 1 rendered) | 0.00 ms | 1 stages : Render : 0.002ms

This is how it looks when done properly (web_vr_dragging):

Surface 0 | 2880x1584 | color 32bit, depth 24bit, stencil 64bit, MSAA 4, Mode: 1 (HwBinning) | 135 192x176 bins ( 96 rendered) | 5.14 ms | 195 stages : Binning : 0.637ms Render : 1.758ms StoreColor : 1.323ms Blit : 0.064ms Preempt : 0.78ms

[CodyJasonBennett]

I don't understand why this should be addressed anywhere else than Quest Browser. Is gl.bindFramebuffer expected to be destructive? The number of calls doesn't change with this PR, only the target. Maybe this can be reverted and instead done by exposing what's currently WebXRManager._getRenderTarget()? It could be instead bound by EffectComposer or OutputPass when rendering to screen if that gives any behavioral benefit.

[CodyJasonBennett]

I now have #26902 which does so. webxr_vr_postprocessing is a good test.

[cabanier]

I don't understand why this should be addressed anywhere else than Quest Browser.

This is unfortunately nothing that the browser can fix. It's a side effect of how mobile/tile based GPUs are designed.

Is gl.bindFramebuffer expected to be destructive?

Yes :-\

The number of calls doesn't change with this PR, only the target. Maybe this can be reverted and instead done by exposing what's currently WebXRManager._getRenderTarget()? It could be instead bound by EffectComposer or OutputPass when rendering to screen if that gives any behavioral benefit.

There are definitely ways to solve post-processing with careful coding and using layers (especially on Quest 3). They require deeper changes to three though.

[CodyJasonBennett]

I just noticed there's a depth/stencil attachment configured in WebXRManager. That would explain your comments. Thanks.

[Mugen87]

Should we still merge #26902 or revert for now?

[CodyJasonBennett]

#26902 reverts the changes to WebGLRenderer with an alternative solution for postprocessing. I believe that addresses this issue, but would appreciate device testing to confirm. I'm also happy to pick in any lost changes to examples and such if you'd prefer to simply revert.

[cabanier]

#26902 reverts the changes to WebGLRenderer with an alternative solution for postprocessing. I believe that addresses this issue, but would appreciate device testing to confirm.

I applied your change locally. It didn't seem to make a difference.

[CodyJasonBennett]

I'm at a loss of what's actually happening then. I've implemented many post-processing systems that worked well as low as Pixel 3a, notably via mipmaps in a similar fashion such as MipmapBlur out of pmndrs/postprocessing. This has included bloom, SSAO, and full volumetric effects. @RenaudRohlinger may be inclined to demo such examples, but I have no direction as for how to continue.

[Mugen87]

Thanks for testing! Okay then let's revert for now.

[cabanier]

I'm at a loss of what's actually happening then. I've implemented many post-processing systems that worked well as low as Pixel 3a, notably via mipmaps in a similar fashion such as MipmapBlur out of pmndrs/postprocessing. This has included bloom, SSAO, and full volumetric effects. @RenaudRohlinger may be inclined to demo such examples, but I have no direction as for how to continue.

I'm sorry to be the bearer of bad news. :-
Stereo foveated headsets require more careful coding than mono screens that are not as sensitive to performance.
It IS definitely possible by re-arranging how things are rendered so you only bind once to the main framebuffer, by discarding depth when needed and by doing postprocessing that doesn't require a resolve.

[CodyJasonBennett]

I thought #26902 would help in that regard, but I'm happy to help if there are other means you find more appropriate.

[CodyJasonBennett]

I'd like to revisit this. I don't think that pinning performance issues with tiled GPUs and multiple passes is fair to this PR nor the issues it addresses outside of postprocessing and incorrect render state. That's not inherent to this PR, but surrounding usage and implementation. I've also demonstrated how performant effects for mobile exist in the ecosystem and not strictly three.js examples which both rely on changes to core. This PR doesn't affect existing apps, but ensures a valid drawing target is bound which has regressed since the introduction of layers. Performance characteristics do not change this way. This should be considered a fix, and I'm happy to remove changes to EffectComposer if that's the only way for this to be considered as such, but I believe that would be largely an oversight.

[cabanier]

This PR doesn't affect existing apps, but ensures a valid drawing target is bound which has regressed since the introduction of layers.

I don't understand why the introduction of layers affects this. If anything, it makes it easier to work around the rebinding. Since almost all of VR uses tiled GPUs, I don't think we should introduce postprocessing unless it can be made performant on mobile platforms.

[CodyJasonBennett]

I don't understand why the introduction of layers affects this. If anything, it makes it easier to work around the rebinding.

I apologize if I'm completely wrong here. I'm going off of memory from https://twitter.com/Cody_J_Bennett/status/1482585611781480448 which I recall working in the old emulator but not on-device. Same observation prompted this PR when I noticed rendering was to the canvas instead of a layer.

Since almost all of VR uses tiled GPUs, I don't think we should introduce postprocessing unless it can be made performant on mobile platforms.

I'm confused as to why we don't give people the agency to even prove otherwise, let alone allow broken rendering state as-is for other cases. I've already demonstrated numerous times this can be done with pmndrs/postprocessing and other linked related work. I also stress that this PR isn't strictly an enhancement to enable postprocessing, but to ensure that renderer.setRenderTarget (at any frequency) does not break further rendering. I've tried to reduce the changes here to entertain this case only, but I believe that to be a disservice to the ecosystem.

[cabanier]

sorry that I sounded a bit blunt.
I too want to have post-processing support in WebXR. However, the hard part is how to make it performant and I hope that we can make changes to three to make it so.
We likely have to be smart about when we render and introduce multiview. We can also wait for WebGPU which will give us more control over the render pipeline so we don't have to be as careful.

[CodyJasonBennett]

I have to repeat my last comment if you insist on performance, but I will not let this PR be a casualty of multiview. If the order of operations matters for its implementation, then it alone should wait for WebGPU.

[cabanier]

You don't need multiview; it would just make implementing usable postprocessing easier...

[CodyJasonBennett]

Unless this introduces a regression for new or existing code, I still don't see a reason to shoot this down. I'm not going to entertain the dialogue around post-processing + mobile since it's both provably untrue (see any documentation or real-world usage attached to this PR) and besides the scope of this PR which I have to stress is wider than strictly that use-case. I'm not happy I have to bend over backwards to reiterate that point specifically, and it's not been particularly regarded at any point despite me mentioning it in at every point.

[BryanChrisBrown]

I'd like to chime in here, this PR would fix a handful of rendering issues we've seen in three.js when used with the Looking Glass WebXR library, is it possible that the oculus multiview requirements could be split into a separate PR or worked on separately from this PR?

My understanding is that post processing doesn't work in WebXR at all right now, and I think making progress towards more powerful devices, desktopVR, Looking Glass etc. would be helpful.