Regression : Persona 5 wrong color issue (NVIDIA only)

[M220]

After merging #4093 PR that includs #4089 PR, Persona 5 shows wrong colors in main menu as can be seen :
Screenshot from persona 5 gameplay on PS3 that is showing right color just like the RPCS3 before merging #4093 :

Screenshot from RPCS3 after merging #4093 ( look at the color of chairs ) :

Edit : My GPU is GTX 1080ti and GPU Driver version is 390.78 ( i think, it's latest anyway because i updated it yesterday ;) )

[AniLeo]

Need information. GPU and drivers.

[M220]

Sorry i forgot it again 😄
Updated information just now

[13xforever]

Can confirm. Win10 1709, 1080 Ti, 390.77 drivers.
Color shifts with animation when you select Config.

f908daf from Jan 19:

95c6ac6 from Feb 12:

To match the video screencap:

[kd-11]

This bug is nvidia-specific. Tagging appropriately

[HerrHulaHoop]

Contrary to what has been mentioned above, this bug was not caused by #4093 but was in fact caused by #4127.

Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz | 8 Threads | 7.89 GiB RAM | AVX
NVIDIA GeForce GT 630M

PR #4093 - Colours are still accurate

PR #4113 (Master just before #4127) - Colours are still accurate

PR #4127 - Colours are messed up

Current master (#4872) - Colours are still messed up

[ruipin]

@al3x on Discord bisected this issue to commit 33bcdd4

Armed with that information, the only hunk in that file that seemed to change behavior (and was not just an optimization) was 33bcdd4#diff-9a62be0c4e83eae188cd568947c4b7a5L50. Reverting this hunk on the latest master (#4970) resolves the issue, as predicted.

I've tracked this down to the RSX_FP_OPCODE_NRM being to blame. Moving that opcode so that it executes the default path is enough to resolve this issue, e.g.:

ruipin@faed15c

I assume this opcode does normalization, though I am not sure why it needs to be clamped. My immediate suspicion is some rounding differences on Nvidia card cause the normalization on one of the dimensions to grow to ridiculous values and overflow as it is assigned to a H register. The clamping makes it so that this overflow does not happen. I'll play around with Renderdoc tomorrow and try to confirm this.

[kd-11]

Its some kind of precision bug on nvidia hardware. Its absurd to need clamp on normalized result which means on nvidia normalize(n) can return nan or inf.

[kd-11]

NRM(0) should and does return Nan. However on Geforce, clamp() affects nans and treats them as -INF instead of leaving them as-is. This causes NaNs to not propagate down the pipeline like they would on PS3. This build fixes this issue https://ci.appveyor.com/project/kd-11/rpcs3/builds/23802725.

[HerrHulaHoop]

Confirmed! Bug is fixed in the linked build.

[kd-11]

Fixed by #5860