rsx/vp: Fix Demon Souls missing graphics

[elad335]

This is the correct decoding of the vertex value here, it affected Demon Souls because it passed an integer of 0 to the vertex shader and then raised it to the power of -1. RPCS3 previously believed that it results in a float of 0 but no it results in float(1)/65535. So when raising it to -1 instead of NaN being produced it now results in precisely 65535. If you want to benefit from this you need to clear the Shaders cache of this game.

Before:

After:

Fixes #4302

[kd-11]

This one is odd. I had already checked the behavior and it seemed to be Nan. I'll have to check where the hwtest got messed up.

[Linear524]

Great catch ! :)
Maybe this could fix missing geometry in other titles as well =D

[elad335]

@kd-11 A test that raises the result to -1 itself for convenience:
Master:
sample finished, the result is infinity.
Ps3:
sample finished, the result is 65535.000000.
Pull Request:
sample finished, the result is 65535.000000.
mainrsxcrap.zip

[Nerboruto]

one word... fabulous !!!

[kd-11]

@kd-11 A test that raises the result to -1 itself for convenience: Master: sample finished, the result is infinity. Ps3: sample finished, the result is 65535.000000. Pull Request: sample finished, the result is 65535.000000. mainrsxcrap.zip

Thanks. Added to the validation suite.
I have hundreds of random tests all over, I checked the vertex one and it seems I did not test some instruction cases such as negative numbers.
That being said, I have some issues with the SNORM16 handling. I'll write some more test cases as well as testing known games using this layout.

[Asinin3]

Make a new issue and link to this. Put renderdoc with it and log