GS/HW: Add ROV based blending/colclip/fbmask/afail for DX11/DX12

[stenzek]

Description of Changes

Not ready for widespread use, here for testing.

• Make dx11/dx12 not break if the gpu doesn't support rov
• Port across to vulkan renderer with fragment shader interlock
• Optimization and address all the TODOs I left in comments

Rationale behind Changes

ROV can be pretty fast in some games. Plus AFAIL handling isn't double pass.

Suggested Testing Steps

None yet.

Fixes Hot Shots FORE Golf/Everybody's Golf menus, Fixes #1499 Fixes #3876 Fixes #3350

[JordanTheToaster]

Fixes the shadows part of #6839

[refractionpcsx2]

Makes Hot Shots Golf worse, I think the condition/problem has actually been inverted, as you can see from the screenshot, the "blotches" which shouldn't be there, are all that's there lol

DX12:

Vulkan:

SW:

Dump:
Hot Shots Golf FORE! ate depth.zip

[refractionpcsx2]

This problem from The Suffering was said to be an Alpha test/Depth problem but I can't remember the condition to what's happening, but for prosperity that doesn't work either

DX12:

SW:

Dump:
SufferingATE.zip

[refractionpcsx2]

Slight update, I need to do "Maximum" blending to get Hot shots to work, and it is fixed like that

[GitHub-Khi]

Slight update, I need to do "Maximum" blending to get Hot shots to work, and it is fixed like that

Can you please check the PAL version of Hot Shots Golf Fore! as well?

Single Frame GS-Dump:
Everybody's Golf_SCES-52582_20221222213239.zip

[refractionpcsx2]

[DonelBueno]

This is huge for games like SSX and Black, which need Full blending for proper emulation in HW.

On Vulkan, the fastest renderer for both games in master, SSX goes from 46 to 64 fps (6x native) and Black from 51 to 91 fps (6x native).

[lightningterror]

DX12 crashes with sharpening enabled (non rov gpu), also noticed this in the logs.
[ 46.0960] C:\GitHub\master-pcsx2\pcsx2\bin\0(248,32): warning X4000: use of potentially uninitialized variable (FxaaPixelShader)

[Uzarkis]

@stenzek - Do you need more tests with the golf games?

[stenzek]

No, and please stop posting/bumping PRs.

[bigol83]

With the latest commits there is a big improvement to some graphics issues i noticed in MGS3 but there are still some kind of random red flashes with DX11. DX12 and every other renderer is good.

Metal Gear Solid 3 - Snake Eater_SLES-82024_20240512044023.gs.zip

[stenzek]

Not really concerned about DX11, I'll probably drop the ROV path for DX11 prior to merging. It's much slower than DX12.

[seta-san]

Come to think about it. Why is dx11 still supported? Didn’t windows 10 ship with dx12?

[JordanTheToaster]

Come to think about it. Why is dx11 still supported? Didn’t windows 10 ship with dx12?

Because not everyone has a new enough card to use Vulkan so they have to use DX11 such as every Intel iGPU pre Xe Nvidia Fermi and Kepler and AMD GCN 3 and older.