Clean up WebXR manager by calling setRenderTarget

[cabanier]

This change changes WebXR so it calls setRenderTarget instead of binding to the framebuffer directly.
This will fix a couple of issues.

In addition, it introduces the ext_multisampled_render_to_texture extension and uses that instead of using renderbuffers.

This contribution is funded by Oculus.

[cabanier]

This change ended up being much larger than I anticipated.
It adds the following to:

• Add support for multisampled-render-to-texture. This extension will be used if the render target is multisampled and the browser supports it. If WebXR provides an depth texture, this extension will be disabled for the WebXR render targer and will fall back to regular multisample.
• Enhanced setRenderTarget so WebXR can pass in its textures or framebuffer. Passing in a texture pr framebuffer will signal to three.js that it doesn't to allocate them and use the provided ones instead.
• More efficient processing when blitting multisampled data. Previously the depth data was always blitted, even if it wasn't used. We now also discard the msaa data after copying.
• Removed the old WebXR workarounds in the state.
• Ensured that the correct depth buffer types are requested. (There were some mismatches that triggered internal copies)
• Set foveation to the maximum value.

I tried out all the vr (and some non-vr) examples with:

• vanilla WebXR
• WebXR Layers without AA
• WebXR Layers with multisampled buffers and a provided WebXR depth texture (the default)
• WebXR Layers with multisampled buffers and an implied WebXR depth buffer
• WebXR Layers with multisampled-render-to-texture and a provided WebXR depth texture
• WebXR Layers with multisampled-render-to-texture and an implied WebXR depth buffer (upcoming default for oculus)

Performance of WebXR Layers with multisampled-render-to-texture and an implied WebXR depth buffer is on par or slightly better than the origin vanilla WebXR implementation.

[davehill00]

Maybe worth adding a log/warning here. How likely is it that you would end up in this state without user error?

[cabanier]

It's not likely, but the author can force a texture if they wanted to.
I'll add a warning.

[cabanier]

@Mugen87 can you also take a look?

[Mugen87]

Yes! I'll try to review the changes in the next days 👍 .

[Mugen87]

Is EXT_multisampled_render_to_texture a WebGL 2 only extension? If so, the following line should be moved to the above if branch:

three.js/src/renderers/webgl/WebGLExtensions.js

Line 73 in 8cb903d

getExtension( 'EXT_multisampled_render_to_texture' );

[cabanier]

@Mugen87 I think I addressed your comments. Do you have any suggestions on how to best test this?

[Mugen87]

Um, it's probably best to merge PRs which apply deep changes to the renderer at the beginning of a month. In this way, we have the complete dev cycle for more detailed testing.

It's definitely a good sign that the E2E tests all pass (although they do no cover all use cases). I've also tested selected examples manually today by downloading your branch. Everything seems to work fine. However, having more time for verification is of course preferable.

[cabanier]

Um, it's probably best to merge PRs which apply deep changes to the renderer at the beginning of a month. In this way, we have the complete dev cycle for more detailed testing.

That's OK :-)
There's a known rendering issue with the current state of the code in three. Should I provide a temporary workaround for it?

It's definitely a good sign that the E2E tests all pass (although they do no cover all use cases). I've also tested selected examples manually today by downloading your branch. Everything seems to work fine. However, having more time for verification is of course preferable.

Thanks! If you know of more examples, I would love for one of us to try them out.

[Mugen87]

There's a known rendering issue with the current state of the code in three.

Do you mind explaining in more detail the rendering issue? I've not seen something unusual when testing the branch.

[cabanier]

There's a known rendering issue with the current state of the code in three.

Do you mind explaining in more detail the rendering issue? I've not seen something unusual when testing the branch.

The version that is currently checked in uses the ext_multiview_to_texture extension. It allows efficient multisampled code but it suffers from the drawback that a certain order of graphics calls cause the depth buffer to be discarded.
This problem will only show up if this extension is supported which is just the oculus browser for now.

We can revert the PR that introduced this extension or integrate this PR which works around that issue.

[cabanier]

I'm still worried that this is changing colors from the old non-layers based WebXR.
Can we verify that this doesn't affect the rendering of non-quest VR browsers?

[elalish]

I think my fix is pretty safe; it's just making the layers and non-layers code paths have the same behavior: #22997

[arpu]

Okay, I can see that indeed the problem is that when entering WebXR mode the shader is recompiled because the output encoding changes from Gamma to Linear. The reason it changes is that the WebXR render target is set to sRGB, but this line sets the texture encoding to Linear: https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl/WebGLPrograms.js#L95

Then this value is picked up by the renderer as the output encoding, but this is apparently incorrect, as the WebXR renderbuffer is exactly like the default renderbuffer and input values need to be converted into sRGB; it's not done for you. I think Renderbuffers need to use different logic for their sRGB encoding than textures use. What do you think @Mugen87?

tested your Patch on oculus quest2 browser with disabled webxr Layers, initial Loaded Textures looks normal now, but new created Textures in webxr session still darker looks like this needs some redefine?

[elalish]

@arpu I assume that's a separate (but probably related) issue. I can't repro it, so I'll let you explore.

[arpu]

if i remove this line https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl/WebGLPrograms.js#L95 it works in webxr view

maybe this needs be sRGB in a webxrSession

[elalish]

It might have to do with how the textures are being created. I thought that line was my problem too originally, but it turned out the fix was elsewhere. Do you have a simple repro URL? And does it also repro on mobile AR?

[arpu]

will test again on next dev build update with your Patch on https://raw.githack.com/mrdoob/three.js/dev/examples/?q=webxr#webxr_vr_ballshooter

[arpu]

@elalish have reproducible simple example at https://raw.githack.com/vrlandio/three.js/vrlandio/0.136.0.vr2/examples/?q=webxr#webxr_vr_ballshooter ( on Controller select it create a new Geometry and new Material for spawn the Ball)

i can reproduce this with chrome and the WebXR API Emulator and on Oculus Quest 2 OculusBrowser with enabled/disabled Layers

[cabanier]

Okay, I can see that indeed the problem is that when entering WebXR mode the shader is recompiled because the output encoding changes from Gamma to Linear.

As I mentioned in the PR, if we change the texture format to be sRGB, we also need to communicate that to the browser when we create the projection layer. Otherwise, the multisampled path that uses renderbuffers is broken.

I also think we should make this lighter rendering an option as it produces output that doesn't match the 2D representation; at least on quest.

[mrdoob]

The render inside VR should be identical to the one outside VR (as it always has been).

[arpu]

i can still reproduce this with https://raw.githack.com/vrlandio/three.js/vrlandio/0.136.0.vr2/examples/?q=webxr#webxr_vr_ballshooter ( all created balls in the webxr session are darker)

and some side note, if i enable webxr navigation in OculusBrowser and transform in webxr to a new Scene all texture are darker

[mrdoob]

@arpu Can't repro. Do you mind recording a video?

[arpu]

@mrdoob https://drive.google.com/file/d/1_r1-Mhy2PHh_iTCAoBIwwb_0Ok-du74m/view?usp=sharing

[cabanier]

@arpu This user reports that r136 fixed their color issue. Does it still happen for you?

[arpu]

@cabanier yes the example use three.js r136, with r136 all initial loaded bevore enter the webxr session looks good but not the created in the webxr Session, but looks like @Mugen87 refactor the decoding #23129 (comment), better wait to investigate after this

[Mugen87]

I can't imagine how the removal of the sRGB inline decode should affect WebXR scenes. I think your issue needs to be investigated/solved separately.

[hybridherbst]

I can confirm that we're also seeing dark/wrong colorspace WebXR rendering with r136, both on Desktop (via Quest + Link) and Oculus Browser on Quest.

[mrdoob]

@hybridherbst Can you share a screen recording from inside the Quest?

[hybridherbst]

@mrdoob

In Browser (no XR):

In XR (Quest via Link):

It seems to be related to using render textures (doesn't happen when not rendering into any); will investigate further and open a new issue if it's not just a mistake with using RTs on our end.

[Mugen87]

Do you mind checking if the latest dev build solves the issue?

https://github.com/mrdoob/three.js/tree/dev/build

[hybridherbst]

Still happens on latest builds.
Seems the issue happens when rendering into a MultiSampleRenderTarget by doing

// initialize RT
this.rtTexture = new THREE.WebGLMultisampleRenderTarget(256, 128);
...
// at any point in time, e.g. update loop
const xr = rend.xr.enabled;
rend.xr.enabled = false;
const rt = rend.getRenderTarget();
rend.setRenderTarget(this.rtTexture);
rend.render(this.rtScene, currentCamera);
rend.setRenderTarget(rt);
rend.xr.enabled = xr;

Before, this wasn't causing any darkening and both the RT and the XR result were correct;
now it seems that the linear encoding of the RT "spoils" the renderer.

Creating the RT with sRGBEncoding fixes the issue – I'm not sure but I think the rendering result shouldn't change when rendering into a RT and then restoring state.

// no darkening anymore
this.rtTexture = new THREE.WebGLMultisampleRenderTarget(256, 128, { encoding: THREE.sRGBEncoding });

[cabanier]

@cabanier yes the example use three.js r136, with r136 all initial loaded bevore enter the webxr session looks good but not the created in the webxr Session, but looks like @Mugen87 refactor the decoding #23129 (comment), better wait to investigate after this

I applied your change to the latest version of three.js and the balls look identical.
Can you try the same?

[arpu]

@cabanier latest master threejs version? 91e4d29

[cabanier]

@cabanier latest master threejs version? 91e4d29

Latest dev version. My top commit is 9daf001

[arpu]

@cabanier Yes this fixed the Problem with loaded Materials looks darker! Awesome

[cabanier]

@cabanier Yes this fixed the Problem with loaded Materials looks darker! Awesome

When you say "this fixed the problem", what change are you referring to?

[arpu]

not sure tested latest dev version 0311f8f

[gonnavis]

I couldn't think of a way to avoid the new (technical) properties of WebGLMultisampleRenderTarget. Maybe we can figure this out at a later point (e.g. with a new subclass of WebGLMultisampleRenderTarget).

I'm really facing severe glitch effect ( after upgrade from r134 to r135/136 ) when using WebGLMultisampleRenderTarget with EffectComposer.
If delete this line of code this.useRenderbuffer = this.useRenderToTexture === false; in three.module.js ( r135/136 ), or using default WebGLRenderTarget will ok.
And feel it's strange that put these logics in and only in WebGLMultisampleRenderTarget.js:

    this.ignoreDepthForMultisampleCopy = options.ignoreDepth !== undefined ? options.ignoreDepth : true;
    this.useRenderToTexture = ( options.useRenderToTexture !== undefined ) ? options.useRenderToTexture : false;
    this.useRenderbuffer = this.useRenderToTexture === false;

I'll try to make a small example.