[3.x] Sort based on camera position when using perspective camera

[clayjohn]

Fixes a sorting bug reported on RocketChat and likely other reported sorting issues.

The core of the problem here is that sorting appeared to be dependant on view angle. This is because Godot is calculating distance based on the distance to the near plane instead of distance to the camera origin. When using a perspective camera, sorting should take place based on distance to the camera origin rather than distance to the camera plane.

This is a small change with big consequences and needs a lot of testing before being merged.

A similar change needs to be made in 4.0 as well (likely in multiple places see: #60253)

Here is how it worked in Godot 2

godot/drivers/gles2/rasterizer_gles2.cpp

Lines 4853 to 4857 in 820dd1d

	if (camera_ortho) {
	e->depth=camera_plane.distance_to(p_instance->transform.origin);
	} else {
	e->depth=camera_transform.origin.distance_to(p_instance->transform.origin);
	}

Here is how it works in Ogre:
https://github.com/OGRECave/ogre/blob/d302dee15b389cb916dfa13b1e4e55cd8cffaa55/OgreMain/src/OgreNode.cpp#L663-L670

The Unity Docs also suggest using the near plane for sorting when using an orthogonal camera and the origin for perspective https://docs.unity3d.com/ScriptReference/Camera-transparencySortMode.html

Drawing of the problem with distance to plane

[lawnjelly]

Makes sense. 👍

Out of interest, what were the arguments pro / against using distance rather than distance squared, I noticed Ogre didn't do this? Greater precision, or does it interact better with the rest of the sorting which expects linear distance? Not that I think it will make a great deal of difference performance wise for the number of objects typically involved, even if doing it per light etc.

I suppose it could lead to some incorrect swapping of e.g. windows that are close and parallel to the viewer. But this kind of sorting is always very rough anyway and subject to all kinds of issues.

[clayjohn]

Out of interest, what were the arguments pro / against using distance rather than distance squared, I noticed Ogre didn't do this? Greater precision, or does it interact better with the rest of the sorting which expects linear distance? Not that I think it will make a great deal of difference performance wise for the number of objects typically involved, even if doing it per light etc.

I suppose it could lead to some incorrect swapping of e.g. windows that are close and parallel to the viewer. But this kind of sorting is always very rough anyway and subject to all kinds of issues.

My guess is that regular distance is the naive approach and squared distance is the optimized approach. Godot has always just used regular distance, I'm not sure there has ever been arguments in either direction. The only thing to consider is consistency with other areas that rely on depth. In Godot 3.x it wouldn't be very hard to switch to squared distance if we wanted, in Godot 4.0 a lot of different systems rely on the distance (Lod, Occlusion, Geometry Fade, Decals, Clusters) so we would have to be very careful.

[clayjohn]

cc @tinmanjuggernaut This PR may be of interest to you. I'm not sure if you continue to have depth sorting issues, but if you do, this may help.

[Calinou]

My guess is that regular distance is the naive approach and squared distance is the optimized approach. Godot has always just used regular distance, I'm not sure there has ever been arguments in either direction.

From my testing, we can use squared distance easily for "hard" cutoffs, but smooth fading should still use standard distance to prevent skewing the fading curve.

[TokisanGames]

cc @tinmanjuggernaut This PR may be of interest to you. I'm not sure if you continue to have depth sorting issues, but if you do, this may help.

Yes, this was an issue if you were standing among two trees with overlapping branches. Turn the camera one way and one set of branches overlaps. Rotate the camera and the other tree pops forward. Does this fix that?

What about depth sorting with transparent objects with MMI? Which is broken/backward last I checked. Does this help that?

[clayjohn]

Yes, this was an issue if you were standing among two trees with overlapping branches. Turn the camera one way and one set of branches overlaps. Rotate the camera and the other tree pops forward. Does this fix that?

Potentially yes. Sorting is no longer camera-orientation dependant so sorting should be stable on camera rotation.

What about depth sorting with transparent objects with MMI? Which is broken/backward last I checked. Does this help that?

MultiMeshInstance doesn't do any sorting of objects within the MultiMeshInstance, so this may still be an issue. This PR won't help sorting within a MultiMeshInstance, but may help sorting between multiple MeshInstances.

[lawnjelly]

Have approved this as we have to test at some point. One additional possibility is we could make this a project setting to swap between the two modes, even if temporarily? But maybe this would be overkill.

It may cause some slight differences in sorting with e.g. parallel alpha blended polys with a sufficiently different AABB. But maybe the benefits outweigh this.

[akien-mga]

Thanks!

[atirut-w]

Does this PR effects transparent object sorting by any chance?

[lawnjelly]

Does this PR effects transparent object sorting by any chance?

Yes, have you seen any regressions?

[atirut-w]

I don't test alpha/beta and PR versions so idk but I'm happy if it will make transparency sorting more accurate

[clayjohn]

I don't test alpha/beta and PR versions so idk but I'm happy if it will make transparency sorting more accurate

This should make transparency sorting more accurate :) You may or may not notice a difference in your project though.