Vulkan: Raster occlusion incorrectly culls objects through gaps

[lawnjelly]

Godot version

4.0 dev df57aa6

System information

Linux Mint 20, Intel HD graphics 630

Issue description

Objects are being culled when they should be visible, in cases where there are small gaps between occluders.

Setup:

What should be seen:

What is seen:

Occlusion buffer:

Steps to reproduce

Create a couple of boxes as occluders and place them with a small gap, then attempt to view a 3rd object through the gap. Vary the viewing angle / distance.

Minimal reproduction project

TestOcclusion.zip

Discussion

I'm just having a look at some situations likely to trip up the raster occlusion, especially edges / joins between objects. I'm guessing this is caused by the rendering of the occlusion buffer not being conservative (or rather the opposite of conservative rasterization 😄 ), i.e. gaps between objects are being joined. This makes occlusion more effective, but produces visual artifacts.

This one can be quite disturbing as objects flicker in and out of view according to the viewing angle, and whether or not a gap is rendered in the occlusion buffer.

[TokisanGames]

On Alpha13 if the gap is wide enough it works fine.

testocclusion.mp4

To trigger it requires two objects occluding something, it doesn't work with one. And the effective gap between them has to be narrow. Here the objects have a wide gap, but because of the angle, the effective gap has made the red object disappear. The difference between left and right is just camera angle, as the red box is flickering when I move the camera. Interesting how different the occlusion mask is.

[Calinou]

Does anyone have pointers to performing conservative rendering with Embree? GPU support isn't a problem here as raster occlusion all happens on the CPU.

[lawnjelly]

Does anyone have pointers to performing conservative rendering with Embree? GPU support isn't a problem here as raster occlusion all happens on the CPU.

Even with non-conservative rasterization (don't know what the word for this is, but only fill if an entire pixel is within poly) you would get gaps between polys within a mesh. So it wouldn't really be a solution, just a part explanation of the problem.

With ray tracing, I guess the aim would be to find if any ray can pass between the two objects, which could be quite tricky. But you could have a stab. If you recorded e.g. the triangle ID hit per pixel, you could calculate a point between two adjacent triangle IDs to trace a ray to. But this might be expensive, and wouldn't work in all cases. I don't know, perhaps there is some literature on dealing with this problem. Maybe you could use the silhouette edges somehow. 🤷‍♂️

Another thing is if you could identify the import edges you could simply increase the ray count in these pixels, which might be sufficient.

[timshannon]

Maybe add a margin value when generating the culling geometry? Scale the occluder down a bit to account for the gap in the rasterization? It'd be better for it to occlude too little than too much and have obvious pop-in like this right? With a user set margin value, it could be tweaked on a per scene basis.

[lawnjelly]

Maybe add a margin value when generating the culling geometry? Scale the occluder down a bit to account for the gap in the rasterization? It'd be better for it to occlude too little than too much and have obvious pop-in like this right? With a user set margin value, it could be tweaked on a per scene basis.

This is a good idea as anything using pre-processing would be great, but two potential snags:

• The best margin to shrink the meshes may be dependent on view distance / angle (a small margin would work up close, but a larger margin would be needed for distance, to ensure that pixel gap)
• Consider if users build a continuous wall out of two objects that abut, this could introduce a gap between the two, through which the AABBs of occluded objects may then be seen, reducing a lot of the occlusion. That is unless you could detect such cases, which might not be trivial .. perhaps by voxellizing the scene or something like that.

[thoced]

Hello,
Using the occluderInstance3d, I found that this problem still existed. I wanted to post an issue but I realize that the problem was already present in 2021. Has a solution been found?

[Calinou]

Has a solution been found?

No, other than tweaking the meshes you use for occlusion manually.

[Calinou]

This was confirmed by @gongpha in https://www.youtube.com/watch?v=Oskd_NK6r08.

To work around this, they made a custom occlusion culling baker with shrinking along normals: gongpha/gdQmapbsp@9d031ad

This should avoid overocclusion issues without increasing the run-time cost of performing occlusion culling.

[Enhex]

Does anyone have pointers to performing conservative rendering with Embree? GPU support isn't a problem here as raster occlusion all happens on the CPU.

Even with non-conservative rasterization (don't know what the word for this is, but only fill if an entire pixel is within poly) you would get gaps between polys within a mesh. So it wouldn't really be a solution, just a part explanation of the problem.

it's possible to make it work by only allowing "pixel overflow" for internal edges.
internal edges can be defined as edges that overlap and can be computed ahead of time during baking.

also it's better to have these gaps and render a correct image slower than render a very wrong image faster (at which point you disable the occlusion culling, defeating the purpose).
you can optimize for this approach by using occluders with big triangles/polygons.
this is what Urho3D did AFAIK and it worked fine.

EDIT:
you can optimize the internal edges solution by processing fully internal triangles/polygons together, so there's no need for "is internal" check.

[Enhex]

if i understood correctly Godot's occlusion culling uses raycasts to create the depth buffer, which is a wrong solution.
if the depth buffer's pixels are larger then the viewport's pixels they can hide things which should be visible.
if they're smaller (unlikely) they can not hide things which should be hidden.
it's only correct if the occluder buffer and viewport buffer have exactly matching pixels. (not practical, requires millions of CPU raycasts)

instead of raycasting against occluders, rasterize the occluders' triangles.
this approach worked for Urho3D. introduction can be found here in the "Optimizations" section.

to help understand it imagine an extreme case in which you have 1x1 depth buffer, with a raycast at the center of the screen.
if the raycast hits anything, no matter how small, everything will be hidden because its pixel covers the whole screen.

[bedardjo]

Hi, this issue has come up in a project I have been working on and I'd like to share some findings.

Firstly, here's a minimum repro project. This is on Godot 4.2.1.

Also, I took some notes when I was reading through the code trying to understand the issue. Sharing them in case they come in handy: notes.

This section may be of particular interest, as it mentions a few variables I was able to tweak to make the issue go away. This info might be useful to somebody with a better understanding of the code.

Cheers!

[unfa]

Godot's occlusion culling uses raycasts to create the depth buffer, which is a wrong solution.

Oh. So the engine does a raycast for every pixel in the occlusion buffer?
Wouldn't raycasting at a lower res inherently miss everything that is between the rays?
Raycasting will always miss "everything between" the rays, unless we do it at full res, will it not?

And there's an option to instead just rasterize the triangles??

Why isn't that used? Wouldn't it give the same result only much faster and with less errors due to rays being sparse?

[yosoyfreeman]

Yes, the RayCasting solution is problematic for this use case. Not only because you loose information between the rays, but because this gaps can result in zones being occluded that should not. In general lines, one wants to be conservative with the culling, which means that rending more than needed is always preferable to rendering less than needed (rendering exactly what is needed is an unrealistic expectation). This can be achieved in multiple ways, but a common approach is to render a depth texture to a lower resolution using the CPU, and then downscaling it multiple times keeping always the brighter/farther value. This results in a much smaller texture in which the "available space" was prioritized, which makes sure (Intentionally) to show a little more than what is actually needed. You also have a really small texture to perform checks against using the screen space position of the occluders. This technique was used in KillZone 3, for example, and seems very similar to what Urho3D does to.

[unfa]

I suppose many other games use the same idea or something based on it.

To me it looks like we're trying to hammer nails using a cucumber.

Maybe instead of covering the cucumber in chainmail, we can use a hammer instead? :D

PS: are there issues with rasterization that raycasting is solving? Or is rasterization not possible to do for some reason?