rsx: Reimplement depth data handling to support depth_float data outside [0,1] range

[kd-11]

This is a large PR and still incomplete but it does many different things that cannot be supported individually without causing issues.
Changes include:

• Separation of depth buffers into distinct formats for fixed point and floating point.
• Removal of depth targets in blit engine. This massively simplifies the logistics of handling depth data in blit engine, but now everything is routed through compute which may be slower. Testing is needed.
• Implement compute jobs to handle conversion to and from 24-bit floating point formats for depth.
• Enables extended depth range rendering for both OGL and Vulkan. On PS3, depth range is unrestricted when rendering into a floating point depth buffer. Games are commonly running with Z values in the tens of thousands ranges.

Remaining:

• Diagnose and fix all the vulkan regressions.
• Black ops regression.
• Image conversion routines for OpenGL. Due to routing depth data in blit engine through compute, accurate conversion routines are required. This does not work at the moment, making many games break when using OGL backend.
• Hardware clip control in vulkan. A trivial improvement, but I need the fallback path confirmed to work on both backends before I finalize this. This will require shader cache invalidation, so I may delay this and submit as a separate PR. No functional improvement is expected from this, just cleaner code.

Fixes #8360
Probably fixes #6855

[kd-11]

If testing, only test with Vulkan until the first pending item is done. Keep an eye out for performance losses as well as any vulkan device lost crashes that may appear. Testing at different internal resolutions is encouraged.

[Josivan88]

Just to be clear, Nvidia users must test using OpenGL (when it is ready), and AMD users must test using Vulkan? or not?

[kd-11]

No, all users should test Vulkan for now. OpenGL support is incomplete right now.

[MarioSonic2987]

Breaks skybox in One Piece: Pirate Warriors in the first level:

Master:

[MarioSonic2987]

Breaks skybox and water in GTA IV:

Master:

[Andryshik345]

Breaks HUD in Mass Effect

PR

Master

[Josivan88]

007 Reloaded in default resolution fixed! (this PR)

RPCS3.log.gz
4K ok too (this PR)

RPCS3.log.gz

[Josivan88]

call of duty black ops 3 suffering from severe slowdowns (Master is OK)
(this PR) RPCS3.log.gz

[Josivan88]

007 Legends solved (default resolution and 4K) (this PR)

RPCS3.log.gz
There are other problems, may or may not be related, like overbright even using WCB

[Andryshik345]

Breaks almost the whole graphics in GT6

Some screenshots

[ncuxonaT]

Prince of Persia 2008

PR - Sky is gone

Master - Sky is present but clouds have z-fighting

[Josivan88]

Unfortunately, it does not solve the Heavy rain flickering problem: (due to the flickering the frame problem was not so prominent in this picture)

RPCS3.log.gz

[Kravickas]

Not regression.
I just thought to drop it here since u are manipulating with depth and something with conversion.

Star Wars: The Force Unleashed II - Vulkan - same broken gfx and errors on master

·E 0:01:24.441170 {RSX [0x00e0d30]} RSX: Unsupported depth conversion (0x84)
·E 0:01:24.441182 {RSX [0x00e0d30]} RSX: Unsupported depth conversion (0x9C)

RPCS3.log.gz

Feel free to ignore it if not enought relevant or too difficult.

[Xcedf]

Not fixes #6855 for both Heavy Rain and MGS4, unfortunately

[Satan86]

GT5 also regressed with this PR

RPCS3.log.gz

[kd-11]

These all look like Z precision problems due to simulated clipping. Easily fixable with vulkan clip control, but I need to support the fallback because not all drivers are guaranteed to support it. If it proves impossible we can just disable control unless Z clip extension is present.

[kd-11]

Broken visuals in most games are caused by the fact that the near plane can be equal to the far plane causing a division by 0. Turns out fixing that bug then exposes several unexpected issue that break even more games requiring a redesign of sorts.

[gamrXerus]

I can confirm this pr fixes the flickering in Goldeneye Reloaded. However it introduces another issue. The screen becomes white, mostly when looking at the sky in the outdoor maps for splitscreen.

RPCS3.log.gz

In addition, Call of Duty Black Ops cannot reach the main menu with this pr, but it can on master.
RPCS3.log

In Call of Duty Modern Warfare 2 your hand and gun is not rendered with this pr, but it shows fine on master

RPCS3.log.gz

[kd-11]

Fixed a ton of problems from the depth-clip emulation. Retest broken games.
Super bloom in 007 series is still there, not sure what is causing it, I'm investigating.

[gamrXerus]

Modern Warfare 2 is now fixed. Black Ops still wont load.

[Satan86]

GT5 Fixed

[ncuxonaT]

Prince of Persia 2008 is now fixed. I don't see any depth related issues.

[gamrXerus]

Black ops still boots for me on master, but I cant get the start menu to come up with this pr on windows 10 or ubuntu 20.04

[Josivan88]

I can confirm what gamrXerus said COD BO still broken (probably COD BO3 too), more details:

Slowdown on intro videos (not in master)
Game is becoming slow until crash (Not in master)
Last PR version crashed my GPU and my screen become black until driver can recover itself (not in master)
RPCS3 crash and become irresponsible, can't even compress the log at the time. (Not in master)
All the thing is so serious that I can't even make an RSX capture
Log and screens, I tried to configure with all the settings equal to each other:

Master

Log
RPCS3.log.gz

This PR
Black screen, in the best situation, almost reach the main menu
Log
RPCS3.log

[kd-11]

Thanks, looks like there is incorrect detection between float and int depth buffer types. I'll look into it.

[Graubek]

This PR repaired sounds in InFamous 2

[Graubek]

Problem with launching InFamous 1
RPCS3.log

[kd-11]

A huge rewrite is required to proceed with the PR without breaking drivers, I'll try and split some of it into smaller PRs and submit those first instead of wrestling a super-PR.

[kd-11]

Base rewrite PR #8769
That PR does not handle the activision games issue (that will come in a later PR). Please test for crashes or other unexpected effects on that PR, including black ops problems.

[kd-11]

Rebased on vulkan bringup work. There should be no more regressions on Vulkan backend, only improvements. If not, please make a detailed report so I can look into it.

[Crimsom13]

Black ops 1 some rendering improvements on models/hands looks to be going definitely in the right direction still some flickering though and broken in 4k green rendering issue.

[Josivan88]

Any news about that? #8360 is basically fixed on vulkan with this as it is.
And any idea when rsx: Depth float bringup [2 of 3] will come to life?

[kd-11]

Blocked by #8714 which I am working on at the moment. That is the part 2 of 3 and is the main remaining task. With this PR (and master as well) OpenGL is unusable.

[kd-11]

OpenGL part is done, so now only bug checking for this remains to ensure nothing has broken.

[gamrXerus]

Tested a few games using the OpenGL backend with this PR. Goldeneye Reloaded wont boot using OpenGL on both Lubuntu 20.04 and Windows 10.

Linux log :
RPCS3.log
Windows log:
RPCS3.log

[kd-11]

The remaining GL bugs are mostly fixed in #8852