rsx: ZCULL rewrite

[kd-11]

Redesigns the zcull system as well as dependent systems such as conditional rendering. Two modes of operation are offered, controlled by a config variable "Relaxed ZCULL sync". Relaxed sync off behaves a bit closer to master and is the accurate option, while on is an experimental option for games that only need the information for conditional rendering. This however, means that the emulator will do extra work as the state of conditional occlusion is uncertain, including vertex streaming, program decoding and compiling, etc which means setting relaxed mode to on does not guarantee better performance in any way. Currently only known to improve performance in one game (RDR) and causes issues such as hangs and even extremely poor performance in other games. This is the main purpose of having this tested, if it does more harm than good, it can and should be removed before merging.
While relaxed mode off is functionally similar to master, it has undergone significant changes and should be tested just as thoroughly.

[isJuhn]

Ratchet & Clank findings:

ToD
	master: 14.5-15.5
	pr relaxed zcull false: 16-17
	pr relaxed zcull true: 14.5-15.5

ACiT
	master: 20-21
	pr relaxed zcull false: 27-28
	pr relaxed zcull true: 27-28

UYA
	master: 35-35.5
	pr relaxed zcull false: 23.5-24
	pr relaxed zcull true: 27-28

[ghost]

Mostly 18-19 here with dips down to ~15. Ryzen 3600/RX580 (RADV) Linux. I'd say with it enabled is maybe giving me about 1fps average more. Only once I saw a brief graphical issue where some rectangles of the screen sort of popped out, not reproducible or consistent.

Skate 3 has missing graphics at far distances that pop in and out, but doesn't seem as bad as regular zcull disable.

Better perf in some areas I tested, but I've had three crashes already, where audio keeps playing but graphic output freezes and RPCS3's render window needs to be killed.

[kd-11]

Relaxed on can cause problems which is why I was asking for testing both on and off to see if its worth keeping. A ryzen cannot keep the graphics pipe fed well enough to see any appreciable difference ingame, but a big difference can be seen in cases where the bottleneck is graphics, such as cutscenes, especially the opening boat cutscene for new game. Both modes show a big jump for me on a 1700 (master: 19, off: 24, on:28) but I need others to verify in case I'm an outlier.

[SpeddieSwagetti]

Sonic Unleashed:
Relaxed ZCULL ON:
33-36
Relaxed ZCULL OFF:
29-32
Master Build:
31-35

Legend of Spyro - Dawn of the Dragon:
Relaxed ZCULL ON:
95-102
Relaxed ZCULL OFF:
94-99
Master Build:
90-96

Ratchet Deadlocked:
Relaxed ZCULL ON:
35-37
Relaxed ZCULL OFF:
36-38
Master Build:
40-41

Ratchet and Clank - Nexus:
Relaxed ZCULL ON:
21-24
Relaxed ZCULL OFF:
20-22
Master Build:
16-18

Skate 3:
Relaxed ZCULL ON:
42-45
Relaxed ZCULL OFF:
47-51
Master Build:
43-46

Ratchet and Clank - Tools of Destruction
Relaxed ZCULL ON:
15-16
Relaxed ZCULL OFF:
15-17
Master Build:
15-17

SPECS:
8700k @ 4.4ghz
2x8GB @3400mhz
NVIDIA GTX 1080

[rajkosto]

Strangely enough, this PR makes MGS4 graphics-hang (sound keeps going) when the RPCS3 Load Game list comes up, if Zcull is not disabled (when its relaxed it happens very fast, less than a second, sometimes immediately, when its normal zcull, it takes a few seconds but still does it)

FPS results when i did manage to load the heaviest ZCULL impacted savegame in mgs4 though:
ZCULL Disabled: https://images.sshnuke.net/2019-12-18_06-46-33_WmI83xct4.png
ZCULL Normal: https://images.sshnuke.net/2019-12-18_06-49-12_tmVEHNhkt.png (same as before rewrite, maybe a little better)
ZCULL Relaxed: https://images.sshnuke.net/2019-12-18_07-14-53_W2MBtiBN0.png (this took a lot of tries to get through the hanging load game menu)

also with relaxed zcull is the first time ive seen this message in rpcs3 in mgs4: RSX: CB chain has run out of free entries!

[kd-11]

@rajkosto Pushed a fix for the deadlock, retest.

[kd-11]

Looks like MTRSX is causing freezing when loading shaders, still investigating.

[kd-11]

Both issues should be resolved now.

[kd-11]

Refactored again and cleaned up. Also added the relaxed zcull toggle in the advanced tab. Hopefully no new issues introduced.

[kd-11]

@isJuhn @SpeddieSwagetti Could you retry the games where master was faster than the new build with relaxed zcull off? The code has been tweaked a bit since the original release and I was unable to reproduce the slowdown in R&C games although the fps does seem to fluctuate a bit in those games anyway.