VREffect: Use skewed frusta to simulate focal length

[njam]

This is more of a question, as I don't fully understand the topic yet.

This article recommends to use two parallel frustums for stereoscopic rendering, and cutting off a bit on the outer side of each image. This would be done so that the two rendered images overlap completely at the focal plane:

I understand the current VREffect doesn't do it, so the focal length is infinity.

Is this correct? Would it make sense to allow configuration of the focal length?
Does anyone have experience how it affects the experience?
cc @dmarcos

[MasterJames]

Yes a focal point is important.
It depends on many factors ranging from subject mater and desired focal point to screen size and distance to eye separation and lens distortion.
http://doc-ok.org/?p=77
Also links to oculus video explaining the complexities and reality of the precision requirements of small close screens.
I believe the cameras should converge on the focal length as it is pulled for each shot. So like eye separation should be calculated in real time for each viewer so to should there focal point be tracked and images generated for the viewers changing perspective. Ideally film uses focus and subject to direct you atention and the stereo cameras are converging there too.
On big screens for audiences you want the images to align on the single screen more.

[tschw]

Definitely interesting! Please go on, and thanks for sharing the link.

The amount to skew the frusta at the clipping planes, as required for the projection matrices, can be determined as follows:

Parallel frusta (shown dotted blue) have the eye distance at any point. Moving them toward each other by half of the eye distance makes the images converge.
Doing so at the desired focal distance we can easily derive the slope of the skew, which allows us to determine the amount to move at any distance.

THREE.PerspectiveCamera already allows to skew the frustum for the purpose of multi-monitor rendering. The docs of the legacy OpenGL functions glFrustum and gluPerpective contain useful information on how to calculate projection matrices.

[MasterJames]

I forgot to mention the guidelines for the forum are bugs and feature requests only. This kind of thing should be asked on stack overflow.

[tschw]

I understand the current VREffect doesn't do it

No. Doesn't look that way.

@MasterJames OP was making a feature request or maybe even proposing a contribution...

[njam]

Thanks for the explanations and links!
I will do some tests with VREffect to see how it can be done and how it affects the experience. I will report back here.
I understand off-axis projection with skewed frusta could be valuable for stereoscopic rendering, so I would suggest to keep this ticket for tracking research and/or implementation. Wdyt, should I change the title?

[WestLangley]

@njam Yes, you can rephrase your title if you want so it sounds more like a suggestion.

[tschw]

@njam How about you just implement skewed frusta in `VREffect, send a pull request and we close the ticket? ;-) As said, it shouldn't really be too difficult...

[njam]

Ok I added some code!
It's based on the following idea:

So the frustum angle is adjusted like this:

angleNew = arctan( (f * tan(angle) - eyeTranslation) / f )

I tested it briefly with a VR goggle and it seems to work fine.
Wdyt, does it make sense?

[njam]

...I have some concerns that changing the left and right angles of the FOV makes this a "toe in" projection. I.e. that the frusta are in fact not asymmetric.
But I don't understand fovPortToProjection() well enough. Who does?

[tschw]

I have some concerns that changing the left and right angles of the FOV makes this a "toe in" projection. I.e. that the frusta are in fact not asymmetric.

It appears to me you actually got it right. Look at fovToNDCScaleOffsetand fovPortToProjection and compare the resulting matrix to the one produced by glFrustum:

The discrepancy we see is because we use slopes (as produced by tan) instead of distances in respect to the near plane as those left and right parameters of glFrustum, but the resulting frustum actually gets skewed. It's a little hard to decipher and the call to transpose() makes it worse IMO, turning what reads as columns into rows...

I wonder whether it makes sense to also nullify the translation along the Y-axis, this way making sure the two images really match in the focal plane.

[MasterJames]

Initially I wondered if alpha prime was there for the device's lens specs as depicted in the later half of this video, (which was more deeply hidden within the link I posted above relating to Oculus Rift).
Maybe it's not needed?
http://www.youtube.com/watch?v=lsKuGUYXHa4

[tschw]

Initially I wondered if alpha prime was there for the device's lens specs as depicted in the later half of this video, (which was more deeply hidden within the link I posted above relating to Oculus Rift).

The WebVR spec does not provide lens information and tells us to render to a rectangular region. I guess lens distortion is dealt with at a lower level. However, we should probably call the method HMDVRDevice.setFieldOfView to feed the information about the modified frusta back.

Some devices may already give us focused frusta, and in this case the focal distance set by the proposed code will be off (sorry, I didn't look close enough). @njam Could you provide the contents of the VRFieldOfView settings recommended by your device?

It seems quite useful to allow the client to get / set the focal distance, for rendering 2D HUDs / scene-dependent enhancement of the 3D experience. However, a focus at infinity clearly does not make any sense and just seems a recipe for a poor experience.

So what we should do is:

• Determine the default focus recommended by the device.
• Use it, unless overridden explicitly by the client or by ourselves.
• Override Infinity with a better default in case there are misbehaved devices.
• Clamp user settings to the min / max range reported by the device.
• Ensure the client gets the actual focal distance (after clamping).

The code structure could be something like:

var userFocalDistance = 0; // means "not set"
var focalDistance = calculateFocus( device, userFocalDistance );

this.setFocalDistance = function( value ) {

    userFocalDistance = value;
    focalDistance = calculateFocus( device, userFocalDistance );

}; 

this.getFocalDistance = function() {

    return focalDistance;

};

// If there are misbehaved devices:
if ( focalDistance === Infinity ) {

    var RelativeFallbackFocus = 0.25; // 0 (near) ... 1 (far)
    this.setFocalDistance( near + ( far - near ) * RelativeFallbackFocus );

}

[njam]

Very interesting! I didn't know that WebVR provides a way to set the FOV.
Your proposition how to handle changes of the focal distance makes sense to me.

I'm using the experimental Chrome WebVR builds, so there's no "real" FOV-data available. This Chrome build provides an HMD device called "Mockulus Rift" with this FOV:

downDegrees: 53.09438705444336
leftDegrees: 46.63209533691406
rightDegrees: 47.52769470214844
upDegrees: 53.09438705444336

(full left eye params)

This means the focal distance is 1.90 meters, assuming my math is correct.

Calculating the focal distance from the provided frustum angles might be a bit tricky. I'm wondering whether and how we should handle the case that they are not symmetric (i.e. right eye's leftDegrees is different than left eye's rightDegrees)? Maybe log a warning?

On a side note the webvr-polyfill uses leftDegrees === rightDegrees. So your suggestion to catch focalDistance === Infinity and adjust the focal distance in that case seems reasonable. I'm wondering if it's good to base it on near and far though, because I could imagine in many cases far is just an arbitrary high number. Maybe we should just set it to 2?

I wonder whether it makes sense to also nullify the translation along the Y-axis, this way making sure the two images really match in the focal plane.

You mean the y-translation of the eyes? I've never seen it not being zero, what does it mean?

[tschw]

Calculating the focal distance from the provided frustum angles might be a bit tricky.

To make things worse: We should use world space distances when getting / setting the focal distance.

I'm wondering whether and how we should handle the case that they are not symmetric (i.e. right eye's leftDegrees is different than left eye's rightDegrees)? Maybe log a warning?

I simply wouldn't bother, guessing the probability to meet a device that recommends asymmetric eyes equals zero.

I'm wondering if it's good to base it on near and far though, because I could imagine in many cases far is just an arbitrary high number. Maybe we should just set it to 2?

Setting clipping planes to arbitrary distances is (sadly common but) very bad practise, and likely to cause depth buffering artifacts. There's more precision for nearby depth values because of perspective division, but some platforms have lower depth buffers precision.

The default of two meters could have been the reason why you perceived a focus at infinite distance. Are you sure your scene was at least two meters deep? ;-)

This means the focal distance is 1.90 meters, assuming my math is correct.

Seems so. I didn't try to make sense of it, but I got the same result taking a different route (pretty much the direct inversion of your code):

skewSlope = abs( tan(leftAngle) * 0.5 - tan(rightAngle) * 0.5 );
focalDistance = eyeTranslation / skewSlope; // ~1.89852

fovXSkewed = leftAngle + rightAngle; // ~94.1598 deg
fovXParallel = 2 * atan( tan(leftAngle) * 0.5 + tan(rightAngle) * 0.5 ); // ~94.1673 deg

[mrdoob]

Very interesting.

I think we should also consider moving all this into a new StereoCamera class in /src/cameras so we can reuse it. At the moment AnaglyphEffect, ParallaxBarrierEffect, StereoEffect and VREffect each have a different implementation of the stereo camera code.

[tschw]

Not so sure for all of this stuff: I think it makes sense to keep the "lobotomic trigonometry tango" in VREffect, as its very much driven by the WebVR API. A StereoCamera will end up much easier to understand when it just translates the (near) projection planes, instead of having to mess with split viewing angles (in degrees), tan and atan all over the place. It would ideally be a factorization and brush-up of the Anaglyph / ParallaxBarrier code, providing improved means of control. These classes pretty much do what I outlined in my first post in this thread.

Interestingly, StereoEffect, despite of doing some calculations based on .focalLength, uses the same projection matrix for both cameras and just translates their positions, which doesn't focus at all. Is it a bug and can we change it to behave just like the others?

OK, let's build a StereoCamera:

When rendering to a 2D display, we usually don't care too much about realistic viewing: In fact, since the viewer focuses on the screen, we often crank up the viewing angle far beyond what would be realistic in case the screen was used as a window into another world. In that case, the viewing angle would depend on the size and the distance of the display and we'd also be concerned about the scale.

Now, with stereo displays we obviously have to care about scale: Our eyes happen to have a physical distance. How does it translate to the units of our virtual world? More precisely, how can we calculate the amount by which to translate the cameras?

Let's assume the kind-of tunnel vision mentioned before can still work to our advantage when using some kind of 3D monitor. Also, the (good, old, "non-lobotomic") viewing angle is known to be intuitive enough to be provided by the client. For HMDs, WebVR provides enough information to derive it automatically.

From the viewing angle we can get the slope of the frustum width, IOW the frustum width at any distance along the optical axis. With a stereo display, the screen is perceived at the focal distance. Given this distance in virtual units and the physical size of the display, we can determine the scale and thus the eye distance to skew our projection matrices.

The physical size of the display is easy to come by for monitors and projectors, and we can set a guesstimated default (e.g. unreliable DPI * resolution * 0.0254 "/m) for the monitor case. Although the physical size of the screen inside an HMD can be determined using a screw driver, the lens system can transform the optical size & position of the screen, which is the information we're really interested in, to pretty much anywhere. However, it can again be derived from the information provided by WebVR, given that we get a focused recommendedFieldOfView (I consider it a severe bug when there's no focus, and I don't think we should work around it just for the polyfill but instead just file a PR that fixes it):

widthSlope = tan(leftAngle) + tan(rightAngle);
widthAtFocus = widthSlope * initialFocalDistance; // ~4.08 m

Summing up the properties of StereoCamera yields:

Property	Type	Unit
near	number	virtual length
far	number	virtual length
focus	number	virtual length
aspect	number	-
verticalViewingAngle	number	degrees
opticalDisplayHeight	number	physical length
physicalPositionOfLeftEye	Vector2	physical length
physicalPositionOfRightEye	Vector2	physical length

Makes sense?

[mrdoob]

Makes sense?

Yep! So StereoCamera would then be used by AnaglyphEffect, ParallaxBarrierEffect and StereoEffect while VREffect would use it's own projection matrix code. Correct?

[tschw]

That's how I would start.

The idea was to finally allow all of the classes to use a central StereoCamera for the matrix code (hence the lengthy feasibility study), so focus steering would be implemented in one central spot. The angle-based calculations required for interfacing with the WebVR API would remain in VREffect.

[mrdoob]

Sounds good!

[njam]

Very clever with using opticalDisplayHeight to determine the virtual size of the screen!
I think it makes sense.

What will physicalPositionOfLeftEye and physicalPositionOfRightEye be used for?

BTW here's the data Google is collecting about their Cardboard viewers:
https://github.com/google/wwgc/blob/master/docs/HELP.md
I don't understand why they collect "Screen to lens distance".

Sorry for my sporadic replies, I'm currently travelling. I will be happy to work a bit on this next week or the week after!

[bhouston]

PS: #6609 creates the skew that is needed for VR effects and is compatible with the VR skew in 3DS Max, Maya, Mental Ray, etc.

[tschw]

Thanks for filing that PR, @njam .

PS: #6609 creates the skew that is needed for VR effects and is compatible with the VR skew in 3DS Max, Maya, Mental Ray, etc.

Interesting! What was the motivation for that compatibility? Given the parameters usually depend on device / installation, does it make much sense to import them?

I don't think that above use cases can easily be tackled based on these parameters. Although they could be used as the low-level interface for setting up the matrices.

[bhouston]

I don't think that above use cases can easily be tackled based on these parameters. Although they could be used as the low-level interface for setting up the matrices.

It is mostly the above. The above formulation is the official formulation for stereoscopic skew based on projection plane offset. A renderer I wrote for stereoscopic effects (used on Mad Max, Jurassic World, Avengers 2...) used the above matrix formulation for skew for pixel perfect matching with other renderer and tool results (sub-pixel perfect matching is required for compositing different partial results together into the final result you see on screen.)

We can add more options on top, but this is the base formulation I would recommend.

BTW Clara.io already supports these parameters via Film Size and Film Offset (the same parameters as Maya exposes.)

[tschw]

I don't think that above use cases can easily be tackled based on these parameters

To clarify, I was talking about the cases for interactive rendering discussed throughout this thread, such as

• keep the focus on an object that moves back and forth,
• render a HUD at the depth where the user won't get headaches,
• do both of the above in a device independent fashion, as required for HMDs, and
• allow reasonable defaults for 3D monitors.

These are anything but straightforward having to mess with "Film Size" and "Film Offset" directly.

BTW I very much prefer these names over anything that contains "projectionPlane", which in fact is a synonym for "near clipping plane" with hardware z-buffering and thus rather confusing. Note that I've been avoiding the term "focal length" for a similar reason; it's defined as the distance of the projection plane (= near clipping plane) from the focal point (of the projection) thus focalLength === near (the typical code to build a perspective transform derives the size of the near plane from the focal length and the viewing angle).

Also note that when writing a software renderer, the projection plane can be decoupled from clipping considerations, IOW the projection plane can be any slice of the frustum orthogonal to the optical axis, so the naming is less confusing in this context:

The projection plane just defines a mathematical frame of reference for the projection:
When raytracing, clipping is pretty much optional: Rays can originate from the focal point and projection occurs from both sides of the plane in this case.
Software rasterizers / hybrid renderers are not restricted to a specific range of integers put into a z-buffer. In fact, they rarely use z-buffers at all - segment buffers are much more powerful for high quality rendering.
In either case, with these degrees of freedom, envisioning a projection plane somewhere in space and deciding what gets projected onto it become two different things.

[bhouston]

I've been avoiding the term "focal length" for a similar reason; it's defined as the distance of the projection plane (= near clipping plane) from the focal point (of the projection) thus focalLength === near

It isn't that big of an issue, but technically I've never actually seen "focal length" be synonymous with near clipping plane, but I guess they could be. BTW Clara.io implements properly focal length and DOF derived from focal length, F-stop etc.: https://clara.io/view/d328427e-70de-491c-bc1f-8c6bc661b74f )

I guess you can have projectionPlane be the same thing as nearClippingPlane although I haven't seen this, but remember that the formulation in the PR I proposed actually just used projectionPlane so it could calcuation a ratio of projectionOffset to projectionPlane, a result that ends up unit less anyhow (thus it actually doesn't matter where those two are located, whether in front or behind the focal point because the divergence ratio will be the same.)

[mrdoob]

Anyone planning on implementing the StereoCamera idea?

Also, should I merge this PR?

[tschw]

Anyone planning on implementing the StereoCamera idea?

Once the shading system stuff is pushed & merged. That is, unless someone else picks it up first.

Also, should I merge this PR?

No - would be breaking. It doesn't consider the device info and puts the adjustments on top (details somewhere above).

[bhouston]

@tschw, just remember as you do this I will want to ensure that it is still compatible with how cameras are represented in Maya, 3DS Max, and all of the production ray tracers (ARnold, V-Ray, etc.) I can share well tested code in that area if you want as I have achieved absolutely perfect camera matching with Maya/V-Ray/Arnold with a full camera parameter set for stereoscopic rendering.

I am okay if your camera setup is a superset of 3DS Max/Maya, etc,... I do not want to limit you. :) Just we can not invent something new that isn't capable of doing what is already the industry standard for stereoscopic camera setups.

[mrdoob]

I just returned to it and some other unfinished Three.js stuff.

Yay!

[tschw]

Moving slowly but moving...

http://tschw.github.io/angler.js/app

Right-click on Shadertoy content to control the camera (W,A,S,D,Z,C,Q,E & mouse to position the camera, Y to lock the Y axis to the global coordinate system).
Arrow Left/Right: FOV angle (behind the convergence plane)
Arrow Up/Down: Convergence depth (FOV stays constant)
Numpad +/- Keys regulate the disparity scale (relative to FOV, since FOV is zooming)

Repo at https://github.com/tschw/angler.js

[bhouston]

This seems neat @tschw , but none of the keys do not seem to do anything for me, and right click just brings up a context menu. I'm on Chrome beta on Windows 10 x64.

[tschw]

Thanks for reporting. Can you confirm you are viewing a .frag file?

I'm asking because there's no right click handler for images and videos and the context menu is quite normal in those cases (there's nothing that can possibly be changed about the camera that was used for shooting the content, except for image level disparity adjustment and aspect correction - but those properties should probably be considered external to the camera).

I get proper pointer lock with Chromium 48.0.2564.23 (64-bit) which seems fairly close version-wise.

[tschw]

/ping @bhouston
Ben, would you be so kind (see above, I guess there's no notification w/o the name)? Should probably mention that the default test cases can be navigated with the buttons on the upper left, just in case.

I would also be super curious to know whether the stereo parameters behave similar to the tools you are familiar with.

Ended up with two decorrelated values to control the stereoscopic effect.

[tschw]

This figure belongs here too

[bhouston]

I did confirm I was viewing a *.frag. Nothing could make it navigate on my machine, maybe it isn't supposed to?

BTW here is battle tested code for creating a stereoscopic matrix that is pixel-perfect compatible with Maya, Softimage, V-Ray and Mental Ray and was used on a ton of major films for their 3D smoke effects:

void createPerspectiveFOV(M44x & mat, bool isVerticalFOV, Scalar fov, Scalar aspect, Scalar nearPlane, Scalar farPlane, V2x projectionPlaneSize, V2x projectionPlaneOffset ) {

    //EC_LOG_INFO("Creating perspective matrix for FOV: " << fov);
    Scalar fovTan = (Scalar)tan((double)degreesToRadians(fov) * 0.5);
    Scalar width, height;

    mat = zero_constant<M44x>::get();

    if (isVerticalFOV) {
        width = fovTan;
        height = fovTan / aspect;
    } else {
        width = aspect * fovTan;
        height = fovTan;
    }

    const float nearmfar = nearPlane - farPlane;

    mat.x[0][0] = (Scalar)1 / width;
    mat.x[1][1] = (Scalar)1 / height;
    mat.x[2][2] = (farPlane + nearPlane) / nearmfar;
    mat.x[2][3] = (Scalar)-1;
    mat.x[3][2] = ((Scalar)2 * farPlane * nearPlane) / nearmfar;
    if( projectionPlaneSize.x != 0 ) {
        mat.x[2][0] = (Scalar)2 * projectionPlaneOffset.x / projectionPlaneSize.x;
    }
    if( projectionPlaneSize.y != 0 ) {
        mat.x[2][1] = (Scalar)2 * projectionPlaneOffset.y / projectionPlaneSize.y;
    }
}

projectionPlaneSize is generally the film aperture measured in units like inches or cm.
The projectionPlaneOffset is the offset of the optical center within that projection plane measured in the same units. (0,0) offset would be a centered/default optical center.

More details here, see H/V Film Aperture, and H/V Film Offset: https://knowledge.autodesk.com/support/maya-lt/learn-explore/caas/CloudHelp/cloudhelp/2015/ENU/MayaLT/files/GUID-35B872B1-840E-4DEE-B80F-F2715B1E8BF0-htm.html

Now a very important caveat, the above is for stereoscopic film, where one uses the same projector (size and position) for both eyes. This actually is different than the VR case were the projector for each eye is distinctly positioned in front of the eye and the FOV could vary to the left and to the right for each eye (because of the positioning of the image in front of the eye.) Thus the VR specifications give out distinct FOV values for top, left, bottom right separately for each eye as well as the eye offset from center -- just follow those directly for best results: https://developers.google.com/cardboard/android/latest/reference/com/google/vrtoolkit/cardboard/FieldOfView#public-constructors

[tschw]

@bhouston

I did confirm I was viewing a *.frag. Nothing could make it navigate on my machine, maybe it isn't supposed to?

No, certainly isn't. Thanks. Hopefully this commit fixes it.

BTW here is battle tested code for creating a stereoscopic matrix that is pixel-perfect compatible with Maya, Softimage, V-Ray and Mental Ray and was used on a ton of major films for their 3D smoke effects

Ah cool!

So that's how the standard renderer interface looks like - low level, per-cam info and the displacement of the cams is in their positions. Actually the quotient projectionPlaneOffset / projectionPlaneSize is all that matters as far as the projection matrix is concerned (the X/Y shear of the matrix along decreasing world Z called skewSlope above). BTW I have a branch here that makes this vector as an input of THREE.PerspectiveCamera together with a number of other improvements (code simplification, proper serialization of everything, getter for effective fov angle respecting .zoom). Should I go for a PR?

This information must of course come from somewhere. In their low level form, these parameters are hard to control, because all values are codependent. I found this page on the Autodesk site you cited. After Effects seems to have similar stereo settings. This is the kind of stuff I'm after in this thread; to allow the properties of a stereo system to change individually - without an influence on each other.

Interaxial distance and FOV must be chosen appropriate for the display setup (load your favorite FPS -stereo not required- and play it on a projector if you want to know how simulator sickness feels like :).

The interesting value to change is the convergence distance, in order to force the focus to a specific depth. FOV can be changed for a zoom effect, but relative to what is sound according to the display setup. It also makes sense to keep the interaxial distance relative to the tangent of the FOV: This way it remains constant in respect to the screen size and zooming in won't yield insane parallax.

Interesting facts to note about FOVx are

• FOVx of one cam != real FOVx; it's partial,
• shearing the frustum distorts the FOVx for each individual cam, and, much more importantly,
• both frusta cross at the convergence plane and the angle between the outer edges behind it define the real FOVx,
• the real FOVx defines what's really visible and is the same for an equivalent mono frustum.

Now a very important caveat, the above is for stereoscopic film, where one uses the same projector (size and position) for both eyes. This actually is different than the VR case were the projector for each eye is distinctly positioned in front of the eye and the FOV could vary to the left and to the right for each eye (because of the positioning of the image in front of the eye.)

Good point. Calibration is described in the MDN article "Using the WebVR API".

After calibration, except for the possible relative scale you mention, it's not so much different: The optical system engineered into the device makes you perceive the screens somewhere else. For the eyes it's still a single image, there's still a common (virtual) screen of some (optical) size at some (optical) distance that both eyes look at. It really couldn't work any other way. The distortion of the lenses is dealt with at driver level and is non-linear, so doesn't affect the matrices.

Thus the VR specifications give out distinct FOV values for top, left, bottom right separately for each eye [...] -- just follow those directly for best results

These angles are by no means fixed (see link above). They are a more human readable representation of a projection matrix minus clipping planes. Their tangents can be plucked in there pretty much directly - that's why the API is like it is.

However, one really shouldn't mess with the FOV too badly. The API provides min/max pairs for all the angles. Also, it seems advisable to use the setters to communicate the changes back to the driver level.

Luckily the changes of these angles are fairly minimal when changing the convergence distance. Unless doing it wrong and accidentally changing absolutely everything, that is :).

[...] as well as the eye offset from center

This one is actually very subtle. A quality device will report not only a translation along x but also along z. Why? Because it allows you to safely scale the stereo system to your scene without messing up all its properties. Just see below. Note that the cameras have to be moved back to resemble the same frustum.

BTW it's currently implemented incorrectly in Three.js here - only .x is used and scaled.

Scaling a stereo system:

[bhouston]

This information must of course come from somewhere. In their low level form, these parameters are hard to control, because all values are codependent.

Here are some other docs of high level setup parameters:

http://docs.chaosgroup.com/display/VRAY3MAX/Stereo+3D+Camera+%7C+VRayStereoscopic

Actually the quotient projectionPlaneOffset / projectionPlaneSize is all that matters as far as the projection matrix is concerned (the X/Y shear of the matrix along decreasing world Z called skewSlope above).

I would strongly advocate a design that leaves projection plane size, projection plane offset around in the camera so that it is compatible with existing workflows - this is what people get from Maya, Softimage, etc. The stereo camera rig that controls the individual cameras can use zeroParallexDistance and interocularDistance and all those other high level parameters to then control the low level eye Camera settings. That generally is how one does it.

The reason why we should not simplify projection plane size and plane offset into a ratio, as you are suggesting, at least at the user parameter level, is that we need to preserve the film size (projection plane size) if we want physical camera DOF effects, rather than magic number DOF effects. This example here from a year ago does true physical camera DOF based on film size, focal length:

https://clara.io/view/d328427e-70de-491c-bc1f-8c6bc661b74f

BTW I have a branch here that makes this vector as an input of THREE.PerspectiveCamera together with a number of other improvements (code simplification, proper serialization of everything, getter for effective fov angle respecting .zoom). Should I go for a PR?

I think it would be great. I had tried to get projectionplane offset/size in as a PR exactly a year ago:

#6609

The interesting value to change is the convergence distance, in order to force the focus to a specific depth. FOV can be changed for a zoom effect, but relative to what is sound according to the display setup. It also makes sense to keep the interaxial distance relative to the tangent of the FOV

The standard name for that in the industry is "Zero Parallax Plane" which is usually a distance (maybe zeroParallaxDistance) one can set and is distinct again from the projection plane to which the projection plane offset is relative.

I would love to see this in ThreeJS proper. :) But I would keep the StereoCamera doing the interocular stuff and the parallax plane while the individual camera just has filmSize/filmOffset or projectionPlaneSize/projectionPlaneOffset if possible to avoid over simplification that moves away from physical cameras. Not sure how to relate this to the WebVR standard so that there were minimal special cases, but it would be great to do that.

[bhouston]

After calibration, except for the possible relative scale you mention, it's not so much different: The optical system engineered into the device makes you perceive the screens somewhere else.

I thought they gave out the different FOVs for left and right because our eyes have different FOVs for left and right -- our eyes regularly handle only partially overlapping views. I was hoping the VR standard was anticipating more immersive displays coming that would have non-perfectly overlapping views:

https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/Champ_vision.svg/640px-Champ_vision.svg.png

But you are saying I was incorrect on that? Okay. I didn't look into it too far. :)

[bhouston]

Here is the Maya stereo camera rig that sets up the individual camera settings:

http://download.autodesk.com/us/maya/2011help/Nodes/stereoRigCamera.html

It seems to have three stereo modes: converted, off-axis and parallel. The standard one I've seen used is off-axis.

For off-axis it seems that these are the relevant parameters:

• interaxialSeparation - scalar
• zeroParallax - scalar

I am a little confuses as to why it lefts you also set the filmOffsets manually, I think that they are actually computed in the rig automatically in the "off-axis" mode and this is a secondary offset.

[bhouston]

But I guess I do not want to be doing the gauntlet thing to you on this issue, and my preferred design isn't that important to me anyhow. So I likely will support your PR if it is an improvement over what we have right now.

[tschw]

@bhouston

I would strongly advocate a design that leaves projection plane size [...]
The reason [...] physical camera DOF effects, rather than magic number DOF effects

Convinces me. I guess this existing code would also better change then - and I guess I could use your help with it:

https://github.com/mrdoob/three.js/blob/dev/src/cameras/PerspectiveCamera.js#L29-L42

https://clara.io/view/d328427e-70de-491c-bc1f-8c6bc661b74f

Odin's bloody hammer... Viking likes :).

@tschw

After calibration, except for the possible relative scale you mention, it's not so much different: The optical system engineered into the device makes you perceive the screens somewhere else. [...]

@bhouston

I thought they gave out the different FOVs for left and right because our eyes have different FOVs for left and right -- our eyes regularly handle only partially overlapping views. I was hoping the VR standard was anticipating more immersive displays coming that would have non-perfectly overlapping views:

https://upload.wikimedia.org/wikipedia/commons/thumb/e/ee/Champ_vision.svg/640px-Champ_vision.svg.png

But you are saying I was incorrect on that?

I'm not!!! Probably should have used "off-axis zoom" instead of "scale" for it to read more clearly. I'm just pointing out that API designers thought about it. Once we have measured reference points that work during calibration, there's a way to deal with it.

Maya [...] seems to have three stereo modes: converted, off-axis and parallel. The standard one I've seen used is off-axis.

Only considering off-axis projections, here.

For off-axis it seems that these are the relevant parameters:

interaxialSeparation - scalar
zeroParallax - scalar

plus FOV, which is not limited to stereo, it's what it all boils down to.

Units should be pixels or fractional size of display for interaxialSeparation and world units for zeroParallax to embrace interactivity. We should expose the current scale to interface with tools.

(EDIT: Probably a BAD idea. Pixels are way better when the window is being resized.)

This way, once configured the values are constant and touching one won't require changing the others because the change has resulted in a new scale of their units.

[bhouston]

Units should be pixels or fractional size of display for interaxialSeparation

I thought they would be world units? Everyone else uses world units for this separation as far as I know.

[tschw]

@bhouston

Maya [...]
I am a little confuses as to why it lefts you also set the filmOffsets manually, I think that they are actually computed in the rig automatically in the "off-axis" mode and this is a secondary offset.

Yes, I was similarly confused by the Motionbuilder settings. Maybe values below update when changing the ones above (?!)...

[tschw]

@tschw

Units should be pixels or fractional size of display for interaxialSeparation

@bhouston

I thought they would be world units? Everyone else uses world units for this separation as far as I know.

This way it's codependent with the FOV angle. When zooming in by narrowing it, we'd experience extreme sudden parallax, for instance (unless of course adjusting it to the new resulting world scale).

When decorrelated from the other values, interaxialSeparation specifies the intensity of the depth cue / amount of parallax caused by depth. It then pretty much becomes a constant of your viewing setup and is the least interesting value to change interactively.

[bhouston]

• We should probably adopt the filmSize, filmOffset terminology because it is easier to write than projectionPlane*.
• Maya's stereo camera class is derived from the camera class, thus it inherits filmSize, filmOffset, fov, near, far, etc. Thus the stereo camera does require all of those that are already on the base camera class.
• To better integrate filmSize and focalLength, we need to figure out how to resolve their dependence on fov. Basically filmSize, focalLength and fov are dependent upon each other. Maybe we make fov and focalLength dependent on each other and hold filmSize constant? I guess that probably makes the most sense. We can then convert focalLength into a getter/setter (dependent upon filmSize and fov) and add filmSize as a new free member variable? That feels right.

[tschw]

To better integrate filmSize and focalLength, we need to figure out how to resolve their dependence on fov. Basically filmSize, focalLength and fov are dependent upon each other. Maybe we make fov and focalLength dependent on each other and hold filmSize constant? I guess that probably makes the most sense. We can then convert focalLength into a getter/setter (dependent upon filmSize and fov) and add filmSize as a new free member variable? That feels right.

Sounds good. Currently, focalLength is not a property either - it's just in that setter which will lose one parameter if we already have the film size - or we keep the interface as-is, apply aspect and have it set filmSize. Any thoughts on this?

EDIT: Been talking about the base revision of my branch. Obviously has a property on current dev...

[bhouston]

Currently, focalLength is not a property either

It sort of is, just not really used consistently:

https://github.com/mrdoob/three.js/blob/dev/src/cameras/PerspectiveCamera.js#L13

it's just in that setter which will lose one parameter if we already have the film size - or we keep the interface as-is, apply aspect and have it set filmSize

Technically filmSize is independent of aspect ratio, it is the size of the digital sensor or actual film, see: https://en.wikipedia.org/wiki/Anamorphic_format And even with full digital pipelines, animorphic is still supported: http://www.red.com/learn/red-101/anamorphic-lenses

RED advocates not using animorphic lenses for capturing aspect ratios that do not fit the film size, but rather keeping the incoming aspect ratio as one projects onto the filmSize just not utilizing the full filmSize for capture, leaving some pixels unused (e.g. the notorious black bars.) This means that while one has a filmSize, there is also an effective filmSize defined by best fitting the aspect ratio. That gets a bit technical though, and not sure how relevant this is.

[bhouston]

The main complication that ThreeJS has is that the render window aspect ratio constantly varies based on window size or phone orientation. I have problems figuring out how to map that to a traditional camera setting where the aspect ratio and film size is fixed.

[bhouston]

Maybe the solution to ThreeJS's every changing aspect ratio is to introduce an effectiveFilmSize getter that does a best fit of the aspect ratio into the specified filmSize -- thus taking the RED camera suggested approach. Then one will not get unwanted animorphic distortion but one can still calculate physically correct DOF and set the camera up using physical parameters without worrying about the user's display aspect ratio.

[tschw]

@bhouston

This means that while one has a filmSize, there is also an effective filmSize defined by best fitting the aspect ratio.

YAGNI maybe?

Maybe the solution to ThreeJS's every changing aspect ratio is to introduce an effectiveFilmSize getter that does a best fit of the aspect ratio into the specified filmSize -- thus taking the RED camera suggested approach. Then one will not get unwanted animorphic distortion but one can still calculate physically correct DOF and set the camera up using physical parameters without worrying about the user's display aspect ratio.

I can't really say I like the extra complexity it adds. I guess we're effectively talking about the aspect of the DOF blur? Would we consider it, even?

If so, how about .filmWidth and .filmAspect. Using the width, so it's directly accessible for stereo calculations. The .filmAspect would be the physical aspect of the film for DOF and used by the .setLens routine (which takes the height, as common in photography, I guess - what would be a reasonable default aspect? default would be 1).

Feels right?

[bhouston]

Well, if we have filmWidth or filmHeight (or a filmSize as a scalar, which defines the max width/height) we do not need filmAspect, we'd just use "aspect" as the aspect ratio.

[bhouston]

I think the technical term for a single dimension film parameter is filmGauge:

https://en.wikipedia.org/wiki/Film_gauge

[bhouston]

I think this works because filmOffset becomes singular as well and it is only the width / horizontal dimension that generally matters anyhow. One can use filmOffset / filmGauge in a single dimension to determine stereoscopic camera skew. I'll be honest am not confident as we invent new things here that they are the right solutions.

[tschw]

@bhouston

I guess the root of our problem might be, that for windowed rendering we have film !== projection plane, after all.

The projection plane ends up on the screen, where the film size (or its aspect) OTOH defines the distortion of an anamorphic projection lens. Both seem very different pairs of shoes.

When specifying FOV in a photographic fashion, however, both do refer to the same length in different units, if you like. In this case we only consider one axis, so it's essentially scalar.

Seems we primarily need

filmGauge

. This also allows us to turn the half-assed .focalLength property into a getter/setter pair.

Well, if we have filmWidth or filmHeight (or a filmSize as a scalar, which defines the max width/height)

(...and if it's the max we'd have to check aspect <> 1 every time we apply it, so it's kinda impractical)

we do not need filmAspect, we'd just use "aspect" as the aspect ratio.

Depends.

As long as we're fine with isotropic DOF blur, yes.

If we want the COC to become an ellipsis of confusion due to an anamorphic projection lens, we do need it (note: Different magnification along X and Y - I know you actually know, but could be someone tries to follow our discussion). Maybe something like lensAnamorphicDistortion -ideally a little shorter- would be a better name than filmAspect, because it leaves us a clue what it's actually about.

We could either leave it out and add it once we actually implement anisotropic DOF, or add it to the camera interface for completeness and also to the code interfacing with Maya and friends.

You brought up the DOF case, so please say something :).

I think this works because filmOffset becomes signular as well and it is only the width / horizontal dimension that generally matters anyhow. One can use filmOffset / filmGauge in a single dimension to determine stereoscopic camera skew. I'll be honest am not confident as we invent new things here that they are the right solutions.

Right. So let's look at our requirements, again. Since stereo seems to be the only use case at this point, I still think it might be just fine to go with the bare shear. But since we already defined filmGauge above, we might as well have it filmOffset.

I think it all makes sense and the import should be cake enough.

[Mugen87]

VRControls and VREffect are now deprecated, see #11301 (comment)