Material system: Implement GPU frustum culling

[VReaperV]

Builds on #1105.

Implement frustum culling in compute shaders for the material system.

The culling works in 3 steps (performed in 3 different shaders):

1. At the start of the frame, in clearSurfaces_cp.glsl all the atomic command counters for the next frame are cleared.
2. At the end of the frame, in cull_cp.glsl every surfaces bounding sphere is checked against the 5 frustum planes (far plane is skipped because we always have it set to { 0, 0, 0, 0 } for some reason; and we set zFar to encompass the whole map anyway) and the corresponding enabled field in the surface commands buffer is set for the next frame.
3. Right afterwards, processSurfaces_cp.glsl goes over batches of 64 surfaces for all of the materials. If a material has an amount of surfaces that is not an integer multiple of 64, it is padded out to be such with fake surface commands (all of their fields are always 0). Each material has a corresponding atomic counter in an array. The indirect commands from each enabled surface command are written into an indirect draw buffer. After each command is written the corresponding atomic counter is increased by 1, and the returned value, added with a static material offset, is used as the indirect commands offset.

Both of these work in groups of 64 because compute threads are launched in groups of 32 (warp, Nvidia) or 64 (wavefront, AMD). The threads that are going past the last surface just return. Additionally surface commands have to be processed in batches of 64 (surface batches) for this reason and because atomic counter arrays can only be accessed with a dynamically uniform integral expression – data sourced from a UBO with global workgroup ID is such, while per-thread data wouldn't be.
Surface culling also requires all surface commands for every surface: surface command here corresponds to a stage in drawSurf shader. The additional ones (set to id: 0) are the "fake" surface commands which are never actually used, but they have to be there because indirection there is not possible since buffer writes have to be in a dynamically uniform control flow.

All of this is double buffered (MAX_FRAMES == 2) and holds information for MAX_VIEWS * MAX_FRAMES views.

Graph of how this system works:

[VReaperV]

I've fixed the culling now. Just need to deal with the memory issue.

[VReaperV]

Also made it work with r_lockPVS so it can be tested now by just doing /r_lockPVS 1 and looking around.

[illwieckz]

How do I enable GPU culling? I tried the branch but a look in Orbit shows me that it still uses CPU culling on my end.

I enabled r_useMaterialSystem but this seems to not be enough.

[VReaperV]

How do I enable GPU culling? I tried the branch but a look in Orbit shows me that it still uses CPU culling on my end.

I enabled r_useMaterialSystem but this seems to not be enough.

Do you have /r_arb_bindless_textures 1? I've disabled it by default because of the perf problems on AMD that we observed earlier. But it's required for the material system to work.
Also, entities still use CPU culling, but you should no longer see R_RecursiveWorldNode() being called if this is enabled.

[VReaperV]

Also, it might crash or something if there are surfaces with more than 4 stages; this can be patched by changing MAX_SURFACE_COMMANDS (both in cull_cp.glsl and Material.h) to a higher number, but a proper fix will require different shader permutations I think.

[illwieckz]

]/r_arb_bindless_textures 1
Unknown command r_arb_bindless_textures 

If I do this it sets the cvar:

/set r_arb_bindless_textures 1

But I see no difference after both this and r_useMaterialSystem are enabled and I did vid_restart, the code still does a lot of R_RecursiveWorldNode calls.

[VReaperV]

Sorry, got the wrong cvar name. r_arb_bindless_texture is the correct one

[illwieckz]

Thanks.

I now get some flickering and lines like this (atcshd):

Warn: 0 15 
Warn: 15 15 
Warn: 30 2 
Warn: 32 2 
Warn: 34 1 
Warn: 35 1 
Warn: 36 1 

And no R_RecursiveWorldNode  call!

So I guess I'm getting this branch running ! 👍️

[VReaperV]

I now get some flickering

I got the same thing... I think there's somehow concurrent operations going on where the processSurfaces_cp is running at the same time as some drawing commands, which is weird since I've set the memory barriers, and doing a glFlush() or using fences didn't help either... Maybe it's just interacting in some weird way with the indirect parameters extension.

and lines like this (atcshd):

Warn: 15 15 
Warn: 30 2 
Warn: 32 2 
Warn: 34 1 
Warn: 35 1 
Warn: 36 1

Ah, that's just some debug output :)

And no R_RecursiveWorldNode  call!

So I guess I'm getting this branch running ! 👍️

Yup! I'm not sure how it's gonna be on performance right now as the rendering shaders have to wait for the compute ones to complete, but it should be better when I make it double- or triple-buffered either way.

[VReaperV]

Fixed most of the flickering issues now and made it double buffered. Some of the flickering issues still remain, but it's probbaly an off-by-1 error somewhere. Also now it crashes on map change again.

[VReaperV]

Made this properly double-buffered now: R_RenderView() will queue views for surface culling, then in RB_RenderPostProcess() it will dipsatch computes for the queued views. I haven't tested that it works with multiple views yet though, but I need to cleanup and fix some things first before that.

[VReaperV]

I'm not getting any flickering now. Also fixed a crash and hopefully the incorrect memory accesses.

[illwieckz]

Excluding the missing fog, I see no visual glitch anymore in ATCSHD, and I don't experience GPU pagefaults anymore.

This is coming to be in good shape! 😃️

Framerate is around 370 fps on medium preset @ 4K with a Radeon PRO W7600 On Mesa 23.2.1 radeonsi. I usually get around 530 fps on master.

[illwieckz]

I get 400 fps on Mesa 24.0.7 (still 530 fps on master).

[VReaperV]

Hmm, I wonder if it's because of bindless textures still... Well, it's also still culling less surfaces right now because the far plane is ignored (we also have it as (0, 0, 0)) and because there's no occlusion culling here yet, so if you're looking e. g. towards one of the sides on atcs it will render all of the surfaces behind walls etc.

[illwieckz]

Yes I test with the default spectator scene, so in ATCSHD it means the whole outdoor and the whole alien base is in line of sight.

[VReaperV]

Yea... I'll make a separate pr later for occlusion culling, that should fix that part :)

[VReaperV]

This works slightly faster for me now after removing some unnecessary branching.

[VReaperV]

Here's a more concise graph of how this culling works:

[VReaperV]

I've also made this work with multiple different views (i. e. portals) and moved defines to GLHeaders.

Surface commands will now use the minimal array size for the maximum amount of stages used on any compatible surface on the map (padded out to be a multiple of 4 for alignment). This required making the cull shader load after the map is loaded and there doesn't seem to be a better solution than that.

[VReaperV]

Frustum culling can now be toggled with r_gpuFrustumCulling.