[3.x] Take resolution into account when setting the max shadow cubemap size (omni light shadow jaggies fix)

[puchik]

#37678 fixed shadow issues for 4.0, but #24587 still occurs in 3.x. Specifically, omni lights used a hardcoded value for max shadow cubemap size.

Screenshots are from the reproduction project in #24587
2048x2048

4096x4096

8192x8192

16384x16384

[clayjohn]

Thanks for taking this on. Fixing this should result in much better shadowmapping for lots of users.

I support the change in theory, but I think your logic is off.

Also, the same change needs to be done for GLES2.

[puchik]

Thanks for looking over this. You make good points and I pushed changes that should hopefully make more sense and added the same to GLES2.

[clayjohn]

I think we need to think through this in combination with the logic for selecting a cubemap here:

godot/drivers/gles3/rasterizer_scene_gles3.cpp

Lines 4812 to 4818 in 5e43484

	for (int i = shadow_cubemaps.size() - 1; i >= 0; i--) {
	//find appropriate cubemap to render to
	if (shadow_cubemaps[i].size > shadow_size * 2)
	break;
	cubemap_index = i;
	}

The largest value that shadow_size can take is 1/2 atlas_size. So this chunk of code can be though of as "give me the largest cubemap you have that is equal to or less than the size of the shadow atlas. This is way too large. This means if we are rendering to a 2048x2048 quadrant (the default first quadrant), this logic would allow us to render to a cubemap where each face is 4096x4096. Previously, the largest cubemap was 512x512, so it looked okay. But we are still being wasteful with resources for smaller quadrants.

For now, I suggest we:

1. Change the bitshift to 3
2. clamp to max cubemap size (or 1024)
3. merge this PR
4. investigate better selection logic so we arent rendering to a larger cubemap than we need to.

[clayjohn]

I made a PR to your local branch last night with some additional changes I made while testing out your PR. Please take a look and let me know what you think.

[puchik]

Oh, somehow I didn't get an email notification for that. Anyway, that's a great addition and I merged it and squashed the commits 👍

[akien-mga]

Thanks!