rsx/vk: Rework MSAA implementation

[kd-11]

Reimplements MSAA texturing to speed up ping-pong fbo usage by games. This causes an excessive amount of GPU<->GPU transfers and an unreasonable amount of context switches between graphics and compute on the same queue with full barriers. The old implementation we had in the past just ignored these complex situations completely as a hack, but that obviously has limits.
This new implementation treats MSAA input as a regular texture input, with the restriction that filtering is not enabled and sampling is implemented in software. This approach completely eliminates the inline graphics->transfer->compute->transfer->graphics cycle that we had for every draw call when rendering some effects at the cost of violating spec in corner cases when self-referencing fbos are found. For cases where this is a problem, we have strict rendering mode which keeps the previous compliant-but-slow behaviour.

Fixes #11642

[jeremybris]

Might fix #7078 . I can't test. I don't have an Nvidia GPU anymore. For anyone with an RTX GPU, at high resolutions and Anti-Aliasing on "Auto", basically what happened was you would have a huge slowdown, 50% or more, when looking at multiple vehicles, so at the beginning of every race.

[kd-11]

A fair point indeed. NVIDIA is disproportionately affected when MSAA resolve and unresolve is required due to hardware limitations. This should improve things for some other titles such as GT HD as well.

[JimScript]

Well LBP has been fixed, but motorstorm still has a large gpu requirement on my RTX 2070 super. I don't think that the problem is related to AA, because I Turned it on and off with no noticeable gpu differences. Strangely, the gpu usage basically halves when a race is paused.

[Yahfz]

Well LBP has been fixed, but motorstorm still has a large gpu requirement on my RTX 2070 super. I don't think that the problem is related to AA, because I Turned it on and off with no noticeable gpu differences. Strangely, the gpu usage basically halves when a race is paused.

Is that a regression though?

[Darkhost1999]

Is that a regression though?

Looks like it was a reply to the might fix.
#7078

[JimScript]

Yes, I was replying that in my experience, this pull does not fix or affect #7078. It might help a little, but I am still gpu bottlenecked at higher resolutions.

[Darkhost1999]

At first got a fatal error. Nothing clearing cache couldn't fix though.
YEEHAW!

RPCS3.log

[hoholee12]

This introduces alpha transparency issue on Gran Turismo 5. Strict rendering mode fixes it but performance suffers.