rsx: Implement Z value snapping to account for precision errors

[kd-11]

Snap resulting Z values to integral stops to try and account for precision errors.
The cause of the errors is actually the inversion of w. Large w values will give very small results that have tiny precision loss. The computed NDC Z is then very slightly outside the range we want (e.g 1.0000001 in a Z range of 0.99-1.0)

Fixes #11669
Fixes #11765

[hvhbot]

Still there

[kd-11]

Interesting. The value I picked solves the issue on amd but not nvidia. This will be messy lol.

[jonibim]

The issue is still present in SEGA Rally

[Linear524]

I noticed that a lot of things are renders fine on amd but have some precision issues on Nvidia...
Maybe there is some global tweak required for Nvidia driver to fix all those issues by one shot ? Like some calculation switch, something similar to denormals flushing to zero, or positive/negative... Or additional Rounding mode... Or Extra Clamping mode + preserve sign...

[hvhbot]

Interesting. The value I picked solves the issue on amd but not nvidia. This will be messy lol.

I have amd gpu(rx 470)

[kd-11]

Interesting. The value I picked solves the issue on amd but not nvidia. This will be messy lol.

I have amd gpu(rx 470)

Odd, the issue is fixed on my 6700XT.
I'll have to just use a larger range.

[jonibim]

Still the same issue on Sega Rally :/

GPU: Nvidia Quadro P1000

[kd-11]

Extended the range, check if the new build is better once its ready.

[jonibim]

The best value that worked for me was 0.00011875. I came up with this value by doing a binary search on a range until I would find something that would give the best results 🤣. Also, any number > 0.0002 fixes the flickering problem, but presents other issues, such as the background of the game looks like it has a low render distance, so that you see the background popping in as you get closer to it. Any number < 0.00011875 reveals the broken sky problem. There probably exists even a better value between 0.0002 and 0.00011875, however at least this number does the trick for Sega Rally

[kd-11]

Hmm, it may be possible to automatically calculate the range. I'll try and infer the range from the Z range.

[kd-11]

Reimplemented using an algorithm to calculate the breakpoint where changes in the Z value have no impact on the Z buffer anymore. This actually works out very close to your estimate, the real value works out to around 0.000119209296656.
This value is dynamic and depends on the scene setup so it should not unfairly corrupt geometry that does not require the fixup.
@jonibim @hvhbot Please retest

[jonibim]

It works great on Sega Rally! 👍

[Xcedf]

Crysis 3 doing well, sky's fixed

[hvhbot]

Issue is fixed

[Jhonny-2005]

Lgtm

[thuru]

@kd-11 please take a look at issue #10955 I believe you are very close to solving it but it seems it still dIsn't solve there. Thanks

[kd-11]

#10955 is a similar bug but it happens much earlier in the pipeline. It will need a slightly different fix.