GS/HW: Overhaul Texture Shuffle detection and improve target sizing/clearing

[refractionpcsx2]

Description of Changes

Overhauls how texture shuffles are detected, adds a new method of texture shuffle for 007 Russia With Love + The Godfather, and improves the sizing of targets and removing old targets when something else is written in their stead.

Rationale behind Changes

Texture shuffle detection is mostly guess work, it's quite difficult to detect if it's being doubled horizontally or vertically. I have improved it by checking the render targets and source sizes and getting a better estimate from them and it seems reliable on the games I have dumps for.

I also made it so targets get destroyed if they are more than a frame old and are having something completely different being written to them if they aren't preserving any data (of a reasonable size), this was required for Midnight Club 3's headlight shuffle detection to work correctly, as they write it over the beginning of an old framebuffer, which is huge.

Suggested Testing Steps

Test general games, including any of the ones listed below.

Things still to do

• Check if I can do this with Direct3D 11/12 now everything else is working properly (I couldn't earlier, it was broken)
• Remove the UI for the Half Screen Offset hack as it is no longer hooked up.

Fixes #3875 007 - From Russia with Love - post effect/half screen fixed
Big Mutha Truckers - shadow now work
Death By Degrees Fog Slowdown - less bad corner stuff, but post still broken
Fixes #3875 The Godfather - post effect/half screen fixed
Peter Jackson's King Kong - less bad corner stuff, but post still broken
Klonoa 2 - Noxious La Lakoosha end cutscene - less blurry/offset
Mercenaries - Playground of Destruction - post effect working without half hack
Stolen - glows on character and lights in background fixed (post effect still broken, however)
Time Crisis 3 - less bad corner stuff, but post still broken

007 - Russia With Love
Master:

PR:

The Godfather:
Master:

PR:

Big Mutha Truckers:
Master:

PR:

[refractionpcsx2]

I might have buggered this up, it might meant to be rg to ba (I forget the logic for this shuffle, it's based around copying to the same group, so R->G or B->A or visa versa), but I don't think I have a test case.

[fabio-couto]

Hello! First of all, I'd like to congratulate and thank you for your work!!! :)

I compiled this new version in my machine to test "From Russia with Love", and I got really excited with results!

I noticed only a very small issue during explosions, a thin horizontal orange line (examples below)

I don't know if is due my settings (I'm using Vulkan). Below my game settings:
[EmuCore/GS] MaxAnisotropy = 16 accurate_blending_unit = 3 texture_preloading = 0 filter = 2 mipmap_hw = -1 UserHacks_SkipDraw_Start = 0 UserHacks_SkipDraw_End = 0 UserHacks_TCOffsetX = 0 UserHacks_TCOffsetY = 0 Renderer = 14 Adapter = NVIDIA GeForce GT 1030 upscale_multiplier = 3 TriFilter = -1 fxaa = true extrathreads = 5 dithering_ps2 = 0 ShadeBoost = false

I'd love to contribute coding something, but my knowledge in this matter doesn't allow me, so I expect this post can help!

[refractionpcsx2]

As it says when you boot the game, you need blending level 4 (high) for the lighting to look correct

[fabio-couto]

As it says when you boot the game, you need blending level 4 (high) for the lighting to look correct

I tested again with Blending levels High, Full and Maximum, and the issue remained for all of them.

I think it's related to Internal Resolution, I was using 3x Native (~1080p)
Running again with Native PS2, there is no issue!

[refractionpcsx2]

As it says when you boot the game, you need blending level 4 (high) for the lighting to look correct

I tested again with Blending levels High, Full and Maximum, and the issue remained for all of them.

I think it's related to Internal Resolution, I was using 3x Native (~1080p) Running again with Native PS2, there is no issue!

okay, can you open an issue then with a GS dump, and we can look at that at some point, it's not really related to this PR :)

[lightningterror]

I'll checkmark as d3d being done as it's identical to vk/gl without barriers, as for the non barrier implementation out of scope for this pr.