[Regression]: GT4 shuffle misdetection heuristics

[RedDevilus]

Describe the Bug

Weird elements that pop up through tracks or flicker.

regression from #9745 (or 1.7.4899)

Reproduction Steps

GT4 run a race I guess

Expected Behavior

Now looks ugly

PCSX2 Revision

v1.7.4919

Operating System

Windows 10 (64bit)

If Linux - Specify Distro

No response

CPU

N/A

GPU

N/A

GS Settings

No response

Emulation Settings

No response

GS Window Screenshots

No response

Logs & Dumps

Unzip:
Gran_Turismo_4_Online_Public_Beta_SCUS-97436_20230815063859.gs.zst.zip
Gran_Turismo_4_Online_Public_Beta_SCUS-97436_20230815060733.gs.zst.zip

Reported on Discord and I checked the dump and this breaks

Previous behaviour:

Master:
Flickery and weird stuff

Hard to see when you change renderer from Hardware to software as it's a single frame:

[refractionpcsx2]

This is nothing to do with shuffles, seeing 1 broken frame when you switch renderer is not uncommon.