Regression: Retro/Grade messed up graphics on Vulkan and OpenGL (#6357)

[benderscruffy]

have tried differant settings and still the graphics are messed up
most of the time it is a black screen with the music and sound fx

RPCS3.log.gz

[AniLeo]

Needs an RSX capture

[AniLeo]

@kd-11 for triaging what kind of graphic issue is this

[Asinin3]

Still an issue, graphics just output garbage on VK and OGL. tried multiple settings and nothing helped. Relaunching game can output different graphics so don't mislead yourself into thinking one setting helped.

No RSX capture because it kept crashing e.g with:

(in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKRenderTargets.h:81[:10], in function as_rtt)```

[nitroos-ai]

It all began here #6357

[kd-11]

Still an issue, graphics just output garbage on VK and OGL. tried multiple settings and nothing helped. Relaunching game can output different graphics so don't mislead yourself into thinking one setting helped. No RSX capture because it kept crashing e.g with:

(in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKRenderTargets.h:81[:10], in function as_rtt)```

This is correct output, just needs the tiled DMA patches I have on my local. Hopefully I'll have them cleaned up before too long.

[Triticum0]

Note: similar issue happens with mercury hg when RCB is enabled. so tited DMA patches will be the needed for that game to be moved to playable.
https://imgur.com/a/vqk4OLr

[Triticum0]

@AniLeo This needs to be reopened as it is the same #6609

[kd-11]

I checked this with RSX detiling enabled (and CPU blit) and the logos and menu now are mostly correct. However, the detiling is not kicking in for some situations when it absolutely should, the same issue observed in some games like NFS Rivals.

[kd-11]

Found it. Unfortunately it's messy. Blit engine writes to texture cache surfaces then RCB ignores them when loading memory leading to the loss (CPU blit off). When CPU blit is ON, the same fails because the CPU path does not support tiling.

[kd-11]

Fixed by #14934 when using CPU blit. The full fix for GPU-only blit is to unify texture and surface cache needs months of work and is a known long-term goal.