rsx: Clamp MSAA sampling weights to avoid clipping

[kd-11]

Fixes #12514

[Linear524]

Great job KD-11 !
Sorry for the offtop, but can this PR fix #8442 ? There is still error with E RSX: Transfer cropped in Y, src_h=810, offset_y=1, block_h=810
Or that issue is unrelated to MSAA function ?

[kd-11]

Great job KD-11 ! Sorry for the offtop, but can this PR fix #8442 ? There is still error with E RSX: Transfer cropped in Y, src_h=810, offset_y=1, block_h=810 Or that issue is unrelated to MSAA function ?

That is unrelated. It's also more of a warning about missing pixels than an error that can cause crashes.

[Linear524]

TLoU and GT6 is still broken with by this PR:
Projection shadows or shadowmaps are screwed in TLoU.
GT6 have some weird green dots in lower right corner.

[Linear524]

[Linear524]

Forgot to write my HW specs:

• SYS: RPCS3 v0.0.28-15306-d41fe80b Alpha | master
• SYS: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz | 14 Threads | 31.83 GiB RAM | TSC: 2.594GHz | AVX+ | FMA3 | TSX disabled by default
• SYS: Operating system: Windows, Major: 10, Minor: 0, Build: 18362, Service Pack: none, Compatibility mode: 0
• SYS: Current Time: 2023-07- 5T23:24:13
• RSX: Found vulkan-compatible GPU: 'NVIDIA GeForce GTX 1080' running on driver 531.61.0.0
• SYS: Firmware version: 4.87

[kd-11]

Open a bug report. The PR has already been merged.