rsx/vk: Implement hardware instancing

[kd-11]

Relates to #15507 (comment)

Instead of a single draw + buffer command to patch constant, we implement hw instancing for sets that only change the constant registers (basically traditional transform instancing). This method is much faster than the fallback that uses a fixed buffer and emits barriers + update commands. Practically quadruples performance on NVIDIA cards in R&C tools of destruction (I don't know other games using instancing).

Also refactors the core RSX thread instead of just piling unrelated crap together. See #15424

P.S - OpenGL implementation will come later, I ran out of steam before I could work on that. Fortunately 95% of the work is already done.

[JimScript]

This update seems to cause large patches of instances to blink out of existence at certain angles in Resistance: Fall of Man, the Ratchet & Clank games seem to still work just fine though:

RFOMShrubBlinking.mp4

[kd-11]

@JimScript Dump RSX capture when it is invisible if possible and another when it is visible. If possible with the most minimal camera movement possible.

[JimScript]

Ok...
Foliage gone:
BCUS98107_20241228112818_capture.rrc.gz

Foliage appear:
BCUS98107_20241228112842_capture.rrc.gz

[digant73]

I confirm the issue on trees. I also see a big fps improvement with lower GPU usage as in the picture below.
In current version, in that scene the fps is about 30 with GPU at 100% while in this PR it is now a solid 60fps at about 40-45% of GPU usage.

[kd-11]

@JimScript @digant73 The issue should be fixed now.

Please compare the new version with the previous one when it builds (or you compile from source) in terms of CPU+GPU load and performance. There shouldn't be any performance regressions unless the image was broken before (e.g missing foliage).

[digant73]

the issue on trees is fixed now with no performance regression. Compared to current rpcs3 release, fps and gpu usage are now better (more fps with less gpu usage) in some places (as in my previous screenshot) where there were fps drops. CPU usage in this PR is the same as in current rpcs3 release (even where the current rpcs3 release is affected by fps drops). Other games (killzone, R3, R2 etc.) seem to be not affected by any performance regression.

[JimScript]

Well it seems like Resistance is fixed and the performance still has the same amazing uplift as it did before.

[digant73]

I see (on my gtx 970) a big improvement in all scenarios with foliage (grass, trees). See the pictures below (first is for current release, second is for this PR). Stable 60fps and lower gpu usage with this PR.

[digant73]

I also confirm the improvements are present on recent GPUs (RTX 4060 in my case). Less GPU usage and more FPS in scenarios with foliage in Resistance Fall of Man

[kd-11]

This PR has already been merged, please don't add comments here. If there's a problem caused by this work, please open an issue ticket instead with logs for investigation.