Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shadows break stereoscopic/3D rendering with post-processing #7732

Closed
brianchirls opened this issue Dec 4, 2015 · 55 comments
Closed

Shadows break stereoscopic/3D rendering with post-processing #7732

brianchirls opened this issue Dec 4, 2015 · 55 comments

Comments

@brianchirls
Copy link
Contributor

@brianchirls brianchirls commented Dec 4, 2015

There are certain cases where post-processing is necessary with stereoscopic rendering, e.g.:

  • cardboard distortion effect when it's not supported natively by the browser in Web VR
  • SSAO

Here's what currently happens in these cases, for each eye:

  1. Set the render target to the buffer that will be post-processed
  2. Set the scissor and viewport to half the screen (either left or right)
  3. When WebGLShadowMap.render gets called, sets its own render target and viewport, but it stores the previously set viewport
  4. When WebGLShadowMap.render is done, it sets the viewport back to half screen.
  5. When WebGLRenderer.render continues, it sets the render target again, but setRenderTarget sets the viewport to the full size of the render target.
  6. WebGLRenderer renders the scene at full size, across both eyes, breaking the output.

Somewhere at or after step 5, we need to be able to set the viewport back to the half screen values set in step 2.

While we're at it, as a bonus, it'd be nice to have a way to only render the shadow map one time. Repeating for the second eye is redundant.

(cc @borismus)

@borismus

This comment has been minimized.

Copy link
Contributor

@borismus borismus commented Dec 7, 2015

Thanks for logging this. Related to some issues folks have been filing against webvr-boilerplate:

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 7, 2015

You're welcome, and thanks for sharing those other issues. I had a hard time believing I was the only one in the world who tried using shadows in Web VR. ;-)

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 7, 2015

Have you guys tried with the dev version? I fixed an issue with shadowmap and viewports the other day. faebb42

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 8, 2015

@mrdoob: Don't know, 'cause I can't get the dev version to run with webpack. self is undefined.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 8, 2015

Oh well...

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 8, 2015

Is that something that's going to be fixed, or should I file a separate
issue for it?

On Tue, Dec 8, 2015 at 11:48 AM, Mr.doob notifications@github.com wrote:

Oh well...


Reply to this email directly or view it on GitHub
#7732 (comment).

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 8, 2015

I don't have experience using webpack. Hopefully someone that uses it figures out what the problem is and fixes it.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 8, 2015

Okay, I'll file an issue and see what I can do to narrow down the problem.
Any idea if there's an issue or pull request that addresses why 'self' was
used in the first place?

Also, can you point to a commit with the changes that affect shadows and
viewport? I could test it separately and see if it solves the problem.

Thx
On Dec 8, 2015 2:12 PM, "Mr.doob" notifications@github.com wrote:

I don't have experience using webpack. Hopefully someone that uses it
figures out what the problem is and fixes it.


Reply to this email directly or view it on GitHub
#7732 (comment).

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 8, 2015

Also, can you point to a commit with the changes that affect shadows and viewport?

Already did that some posts ago...

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 8, 2015

So you did. Sorry, I wrote that email from my phone and didn't see it. I'll take a look.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 8, 2015

@mrdoob I'm afraid that commit did not fix the bug. The problem seems to be in step 5 (see above), which happens in WebGLRenderer.render.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 9, 2015

I see... Do you mind providing a jsfiddle/example I can use to fix the issue?

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 9, 2015

Here you go: https://jsbin.com/kesodi/edit?js,output

I'm actually kinda curious to see what this'll look like in cardboard when it's working.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 11, 2015

Shadows seem to work fine for that example on my Nexus 5. What's the issue?

screenshot_20151211-010417

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 11, 2015

That's supposed to be stereoscopic. i.e. there should be one monolith in each eye. Try turning shadows off and then see what the output looks like.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 11, 2015

Oh! LOL. Of course... 😅
OK, I'll continue investigating 😊

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 11, 2015

;-)

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 11, 2015

Ok, instead of going top to bottom I'm going bottom to top and see when it breaks.

http://jsfiddle.net/ebayetvL/

This works so far... I guess now we need to apply the Cardboard effect?

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 11, 2015

Yeah, the problem only shows up if you have a post-processing pass. The Cardboard distortion effect as included in my example will do it.

The easiest way to debug that is to use Chrome Dev Tools and enable "Device Mode". The Cardboard distortion effect has a build-in whitelist of phones that it works on, because it needs to know the physical screen dimensions. So you need to emulate the UA string of one of those devices to make it work. Nexus 5 will do, or you can see the full whitelist here.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 11, 2015

Can we add the distortion effect to my jsfiddle instead? I would prefer not to have to dig inside webvr-boilerplate.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 12, 2015

I think any post-processing effect should do it. I tried modifying your jsfiddle, but I'm hitting my head against the wall a bit here to make it work since I'm not super familiar with post-processing in three.js. (I usually roll my own.)

The key point here is that the output of a stereo effect needs to be rendered to a render target, rather than directly to the screen. And then that render target needs to be passed through another effect and then rendered to the screen.

Another thing that might be causing some trouble/confusion here is that neither VREffect nor StereoEffect allows passing of a render target to the render method. (I was using a custom version of VREffect on some of my own projects that did that, and it's a pretty simple patch.) So webvr-boilerplate is forced to call setRenderTarget before calling render, which doesn't always work since render sets it again.

@mrdoob

This comment has been minimized.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 19, 2015

Wow, looks like you put a lot of work into this. Thanks. I will take a closer look at the code when I have more time. I would like to try this out with another post-processing effect, like maybe SSAO, just to be sure.

It might make things a little easier and more stable if the render method for CardboardEffect, VREffect and all the other stereo effects accepted a third argument of a renderTarget, just like the method on WebGLRenderer. I can do this myself and make a PR if you like.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Dec 19, 2015

It might make things a little easier and more stable if the render method for CardboardEffect, VREffect and all the other stereo effects accepted a third argument of a renderTarget, just like the method on WebGLRenderer.

I actually wanted to remove renderTarget and forceClear from WebGLRenderer.render() 😁

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Dec 19, 2015

Interesting. well, if it's robust and consistently works by setting renderTarget outside the render method, even with shadows, than that's great too.

@FreakTheMighty

This comment has been minimized.

Copy link

@FreakTheMighty FreakTheMighty commented Jan 15, 2016

@mrdoob I'v modified webvr-boilerplate with the changes you suggested in #7732 (comment) and grabbed the latest from dev, but I'm still only seeing one eye rendered. The effect is the same as you saw here #7732 (comment).

The Cardboard demo you have in dev looks good on desktop, but I can't get it to run on my Nexus 5. I get a very playful, "Rats! WebGL hit a snag."

Can anyone else on this thread confirm the fix on mobile? Also, @brianchirls did you ever get webvr polyfill working with this fix?

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 15, 2016

The Cardboard demo you have in dev looks good on desktop, but I can't get it to run on my Nexus 5. I get a very playful, "Rats! WebGL hit a snag."

Yes, that's because the shadow code is a hack at the moment (fixing it right now). Try removing the shadow from the example.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 16, 2016

@FreakTheMighty No, I'm still having trouble with it. Will try to debug over the weekend and report an update.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 16, 2016

@mrdoob I can confirm what @FreakTheMighty said a couple days ago, that only one eye is being rendered. And now with or without shadows.

Here's what's happening (for each eye):

  1. webvr-boilerplate sets the viewport (and scissor) to half the screen
  2. webvr-boilerplate calls WebGLRenderer.render with a renderTarget
  3. WebGLRenderer.render calls setRenderTarget, which then sets the viewport to the full size of the renderTarget, overwriting the previous call to set the viewport.

Also, WebGLRenderer.setViewport does not take into account the pixel ratio. It looks like that's only considered inside of setRenderTarget when the renderTarget is null.

@mrdoob

This comment has been minimized.

@borismus

This comment has been minimized.

Copy link
Contributor

@borismus borismus commented Jan 16, 2016

The point of webvr-boilerplate is that it's compatible with WebVR via
VREffect. This is not true for CardboardEffect.

So it will take a bit of thinking to integrate @mrdoob's faster
implementation into
https://github.com/borismus/webvr-boilerplate/blob/master/src/cardboard-distorter.js
.

On Sat, Jan 16, 2016 at 8:32 AM Mr.doob notifications@github.com wrote:

Yes, webvr-boilerplate needs to be updated.

They should use these files as reference:

http://rawgit.com/mrdoob/three.js/dev/examples/webgl_effects_cardboard.html

http://rawgit.com/mrdoob/three.js/dev/examples/webgl_effects_cardboard.html?0
(no shadows)

http://rawgit.com/mrdoob/three.js/dev/examples/js/effects/CardboardEffect.js


Reply to this email directly or view it on GitHub
#7732 (comment).

@jzitelli

This comment has been minimized.

Copy link
Contributor

@jzitelli jzitelli commented Jan 16, 2016

@brianchirls @FreakTheMighty if you are having problems, you can try also my quick hack of CardboardEffect into webvr-boilerplate: http://jzitelli.github.io/webvr-boilerplate/cardboardeffect/index.html

Also be aware that the eye separation for the StereoCamera which is currently in use for CardboardEffect is computed based on this formula: stereoCamera.focalLength / 30 * 0.5, that was throwing me off for a bit.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 16, 2016

@borismus @mrdoob: We should probably try to coordinate a bit here. If we can have at least a branch of boilerplate with the r74dev implemented, we'll be able to test to make sure it's working. I'd hate to see r74 get released only to find out later about some VR-breaking bug in it that won't be patched until the next release.

@mrdoob: do we have a release timeline for r74?

Thanks, everyone.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 16, 2016

@borismus: If you'd like some help with that integration, I'm happy to do what I can. Find me over at the webvr slack.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 16, 2016

Also be aware that the eye separation for the StereoCamera which is currently in use for CardboardEffect is computed based on this formula: stereoCamera.focalLength / 30 * 0.5, that was throwing me off for a bit.

Would you suggest a better default?

@jzitelli

This comment has been minimized.

Copy link
Contributor

@jzitelli jzitelli commented Jan 17, 2016

@mrdoob it seems to me that the projection matrices set up in StereoCamera are suitable for stereo images that would be viewed through 3D glasses with non-distorting lenses, glasses which don't fill your whole field of view and the goal is not necessarily to make the viewer feel immersed in the 3D environment (the 3D image will not be filling their whole field of vision if non-distorting lenses are being used), e.g. creating images for a 3D movie that would be watched in a movie theater.

For VR, where we view the screen through distorting lenses, the analysis in http://paulbourke.net/stereographics/stereorender/ would not apply, since the paths to object / screen / eyes are no longer simple lines... And we want the virtual camera separation to match the physical separation of our eyes / the viewer lenses... that is my understanding, I'm not certain if I am correct though :)

My suggestion is to continue using VREffect (which implements the projection matrices based on the given eye separation and other factors), but augment it with the option of rendering the left/right images to distortion meshes as in CardboardEffect. This is what I'm attempting, I'm happy to collaborate/coordinate as well! I'm subvr on webvr slack.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 17, 2016

VREffect uses the projection matrices that the browser gets from the HMD in use, so VREffect works only when WebVR is available. For CardboardEffect we need to compute our own.

Anyway, have you tried to just increase the fov of the camera?

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 17, 2016

My concern here is that setViewport and setScissor do not work as documented, since whatever values you call them with will be ignored as soon as you try to render to another render target...and because they ignore the pixel ratio.

If you'd like to change the way those methods work, then so be it, but it should be reflected in the documentation and they should be replaced with a way to accomplish the same thing.

If there were a clear, functioning way set those properties, then I'm sure @borismus et al could find a way to make the polyfill work. And I'm concerned this issue may cause other problems we're not seeing yet.

@jzitelli

This comment has been minimized.

Copy link
Contributor

@jzitelli jzitelli commented Jan 17, 2016

@mrdoob sorry, I think I'm going off track / misleading by talking about projection matrices... it's the eye / camera separation that I'm concerned about - I don't think this should be a degree of freedom that is indirectly controlled via the focalLength property of PerspectiveCamera. I think it should be set according to the Cardboard lens separation and the scale of the three.js scene. Cardboard V1 lens separation: 0.06 meters, V2: 0.064 meters (according to info in webvr-boilerplate device-info.js)

My suggestion about augmenting / continuing to use VREffect was more focused on the effort to port the distortion mesh back into webvr-boilerplate. I can definitely see the value in also having CardboardEffect which does not rely on WebVR API or the WebVR polyfill to be used.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 19, 2016

My concern here is that setViewport and setScissor do not work as documented, since whatever values you call them with will be ignored as soon as you try to render to another render target...and because they ignore the pixel ratio.

I fixed this last month already (check my Dec 19 comment).

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 19, 2016

@jzitelli Ah, I think I understand now.
So you're suggesting to change var eyeSep = focalLength / 30 * 0.5; to var eyeSep = 0.064;?

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 19, 2016

@mrdoob: I'm looking at your Dec 19 comments, and I don't see a solution. Perhaps you could clarify?

If you're talking about calling setSize, I can't get it to work and don't see how it could. We still need the full width of two eyes in the distortion renderTarget, so changing the size doesn't help, and the horizontal offset is still overwritten anyway.

Again, setViewport and setScissor are now broken and if they are not going to be fixed or if they are only for internal use, then that should be reflected in the documentation, along with an explanation of an alternative approach.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 19, 2016

Please hold. I was confused here, because the links to the code you posted are wrong now that other changes have made to the code.

It's here now:

var width = _renderTarget.width / 2;
var height = _renderTarget.height;
_renderTarget.scissor.set( 0, 0, width, height );
_renderTarget.viewport.set( 0, 0, width, height );
renderer.render( scene, _stereo.cameraL, _renderTarget );
_renderTarget.scissor.set( width, 0, width, height );
_renderTarget.viewport.set( width, 0, width, height );
renderer.render( scene, _stereo.cameraR, _renderTarget );
renderer.render( _scene, _camera );

Let me see what I can work out and then I'll get back to you.

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 19, 2016

@brianchirls http://rawgit.com/mrdoob/three.js/dev/examples/webgl_effects_cardboard.html shows that the issue you reported (stereo breaking when using shadows) is now fixed.

@brianchirls

This comment has been minimized.

Copy link
Contributor Author

@brianchirls brianchirls commented Jan 20, 2016

@mrdoob Yes, I understand. I just want to be able to confirm for certain that it's working in a general case. There are some other smaller issues that this is turning up. If you prefer, I can file them separately.

I'm going through your code and am almost able to replicate your solution with my own fork of webvr-boilerplate. There is one snag that maybe you could help me with? In my code the right eye is doing a clear of the full render target, obliterating everything that was just rendered for the left eye. What are you doing to prevent this from happening in your example?

Thanks.

@jzitelli

This comment has been minimized.

Copy link
Contributor

@jzitelli jzitelli commented Jan 21, 2016

@mrdoob yes, that's really all I was trying to say, heh

@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 21, 2016

@jzitelli apart from the eye separation, all the rest of the code (projection matrix computation, ...) looks good to you?

@jzitelli

This comment has been minimized.

Copy link
Contributor

@jzitelli jzitelli commented Jan 24, 2016

oh sorry, i think that value should be halved (0.064 is the total distance from lens center to lens center, i.e. translate one camera 0.032 left, the other 0.032 right - using the variable name eyeSep there is misleading, maybe call it halfEyeSep?)... the rest looks ok to me, but I'm not an expert, forget what I said before re: projections... I am taking some time to research and understand how the cameras should be set up for Cardboard, maybe others here can give better feedback.

I think it would be good to set up a benchmark scene (e.g. something like the Oculus configuration scene - a desk with various common objects, it provides a good subjective test of how immersive / correctly scaled the VR is).

mrdoob added a commit that referenced this issue Jan 24, 2016
@mrdoob

This comment has been minimized.

Copy link
Owner

@mrdoob mrdoob commented Jan 24, 2016

@jzitelli fixed! 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.