Billboard mode is wrong in VR

[goatchurchprime]

Godot version:
3.2.2.stable

OS/device including version:
Windows 10, GLES3

Issue description:
Billboard mode is useful for putting names/labels above objects and characters in VR. Unfortunately it spins around on its axis whenever you move your head. If you rotate your head from left to right the mesh rotates away from you in the other direction. If you tip your head from side to side the mesh rotates the same way so that it is not aligned with the world Y-axis.

The documentation says when "Billboard mode is enabled, the object's -Z axis will always face the camera." This isn't true: it actually faces parallel to the camera's Z axis, not towards the camera -- although these directions happen to be one and the same in non-VR (no one is picking up their PC monitor and swinging it around the room).

The issue was controversial when I mentioned it in the discord channel because the Y-alignment is true to the camera even when it is wonky on the horizon (see picture). However, I have found that this problem was reported as a bug in Unity and fixed in 2017, so it needs consideration before it is dismissed.
https://issuetracker.unity3d.com/issues/vr-terrain-tree-billboards-position-gets-offsetted-by-vr-headset-position
https://unity3d.com/pt/unity/beta/unity2017.3.0b8

Steps to reproduce:
Simply enable Any billboard mode on any mesh texture in VR

Minimal reproduction project:
I don't know which platform it is best to make the reproduction project on. But take any of the VR demos like this one https://github.com/aaronfranke/godot-GUI-in-VR and find any the textured mesh (GUIPanel3D/Quad in this case), go to material/0 and enable Billboard Mode in the parameters section.

[TheDuriel]

XR: Improved VR Terrain Trees: fixed billboard trees rotating all the time as viewer moves in VR, better transition of 2D to 3D trees, fast "sort-independent" trees with alpha-to-coverage blending and MSAA , billboard texture aliasing improvements

Their issue was specific to the billboard implementation used for trees. And seems to have been solved by interpolating the billboard, instead of it being pure. (as it is in Godot.)

[BastiaanOlij]

Standard billboarding works only on desktop, VRs stereoscopic nature makes it easy for the brain to interpret the billboard being placed unnaturally. Billboards are aligned so Z points backwards. In stereo that means that when you move your stereoscopic depth perception makes your brain think the billboard rotates unnaturally.

In VR the best option is to orient the quad to face the player. You can't solve this in the shader because it will orient the billboard slightly different for each eye, which would really screw things up (though one of the changes I've been planning for Godot 4 is to provide more info to the shader about the fact we're rendering stereo cameras so you could find out what the cameras origin is instead of only have access to the eye's positioning).

To solve it for now, simply do it in code on the meshinstance:

func _process(delta):
    var adjusted_position = global_position_of_player
    adjusted_position.y = global_transform.origin.y
    lookat(adjusted_position, Vector3.UP)

That will look correct in VR and do exactly what you want it to do. You just need to provide the position of the player to your script somehow.

[Calinou]

@BastiaanOlij Thanks for providing this code! That said, it's look_at, not lookat 🙂

[BastiaanOlij]

@BastiaanOlij Thanks for providing this code! That said, it's look_at, not lookat 🙂

Details details :) I did write that from memory :)

While i think it's worth having the camera position in the shader at some point (as the view matrix contains the offset of the eye) so we could do this in shader, I don't think we should add any of this logic into the core engine.

While needed for VR this approach, even in shader, is slower then the current billboard approach and seeing the majority don't need this they aren't going to be happy when we introduce overhead for an edge case. Especially when people who do need this for VR can fix it for themselves.

I'll see if I can find some time to add such a shader to godot-xr-tools so people can just use this. It would require a shader param for now to communicate the player position but that's acceptable.

[goatchurchprime]

I did warn that it was controversial. To me the OpenGL billboard feature feels like a hack that happens to have useful applications. Then in VR we want those useful applications, but the hack does not work because our display-screen is physically moving around.

The look_at() function work-around doesn't scale and can cause juddering images when it doesn't sync perfectly with the frame-rate (I find you have to use _physics_process() for these world positioning adjustments). It's not a stereoscopic issue -- the label in the picture needs to be aligned with the horizontal plane, not 45 degrees tilted relative to it.

I cannot predict what the consensus on this issue will be as to how -- or if -- it gets fixed. However I am sure the feature as it works now is of no use in VR. Maybe the docs could include the line "Billboard mode not recommended in VR", which would save the next person from trying to make it work.

[BastiaanOlij]

Indeed always a conundrum, so much in performance optimisation is trickery, illusion, and it breaks in certain conditions. People won’t be happy if you go for a less performant option just to deal with an edge case. Adding a note in the documentation that billboarding can lead to issues in VR due to stereoscopic effects could be a good idea but it’s far from entirely useless, for many particke effects the optical anomoly is not noticable I’ll need to experiment with your use case some more because there is a way to do this right, even if you need to do a little more in code. Then we can see what we can add to godot to make this available

On Fri, 28 Aug 2020 at 10:00 pm, Julian Todd ***@***.***> wrote: I did warn that it was controversial. To me the OpenGL billboard feature feels like a hack that happens to have useful applications. Then in VR we want those useful applications, but the hack does not work because our display-screen is physically moving around. The look_at() function work-around doesn't scale and can cause juddering images when it doesn't sync perfectly with the frame-rate (I find you have to use _physics_process() for these world positioning adjustments). It's not a stereoscopic issue -- the label in the picture needs to be aligned with the horizontal plane, not 45 degrees tilted relative to it. I cannot predict what the consensus on this issue will be as to how -- or if -- it gets fixed. However I am sure the feature as it works now is of no use in VR. Maybe the docs could include the line "Billboard mode not recommended in VR", which would save the next person from trying to make it work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO262PDZ2DCVE2YNDP6UCTSC6L5HANCNFSM4QNJFG2A> .

-- Kindest regards, Bastiaan Olij My personal profile on Facebook <https://www.facebook.com/bastiaan.olij> My professional profile on LinkedIn <https://www.linkedin.com/in/bastiaanolij/> My profile on Twitter <https://twitter.com/mux213> My tutorials on Youtube <https://www.youtube.com/BastiaanOlij> My code repositories on Github <https://github.com/BastiaanOlij>