rsx: Shader decompiler improvements

[kd-11]

Gathers aliased registers when writing to output for ROP operations. This phenomenon was observed when messing around in real hardware a while ago. Technically you can do perfectly ok rendering without ever touching the output registers if the corresponding half-registers are written to (and vice-versa).

Closes #15651

[ghost]

Are testers still needed? I could help if so, just lemme know what to do.

[kd-11]

Are testers still needed? I could help if so, just lemme know what to do.

@BrentPeterson2028 Just try out random games and check that they still work.

[Darkhost1999]

I tested Far Cry 3 and I seem to be getting a bunch of fatal errors
Master Working.log

E RSX: 
F {RSX Worker 10} SIG: Thread terminated due to fatal error: Failed to compile fragment shader
(in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKProgramPipeline.cpp:30[:25], in function 'VkShaderModule_T *vk::glsl::shader::compile(void)')
E RSX: ERROR: 0:696: 'assign' :  cannot convert from ' temp highp 2-component vector of float' to ' global highp 4-component vector of float'
ERROR: 1 compilation errors.  No code generated.

RPCS3-PR.log
This all occurs at game boot before I even see an intro video.
Here is a test with a clean cache
RPCS3-PR.log

• Games with error:
  LEGO Harry Potter: Years 1-4
  RPCS3.log
  Elder Scrolls Oblivion
  RPCS3.log
• Working Games:
  Red Dead Redemption
  Call of Duty Black Ops
  Uncharted: Drake's Fortune

[kd-11]

Thanks.

[LucioXerus]

I tested 4 Games on fedora.
007gereloaded.log.gz
NFSCarbon.log.gz
Tekken6.log.gz

1.Call of Duty Black Ops splitscreen- works the same as before with the black flickering on the textures, so no change. It is hard to do an RSX capture of this because it is so quick.
2. Tekken 6- I get no audio with this pr for this game.
3. Need for Speed Carbon- The game does not boot with this pr. Compiling modules doesnt seem to work properly.
4. Goldeneye Reloaded. I get no ingame audio with this pr. The intro cutscenes play audio but the game does not.

[kd-11]

I've reworked the way writes are tracked, it shouldn't crash anymore. Please retest.

[Darkhost1999]

Same error, Far Cry 3 tested with a clean cache
RPCS3.log
Lego HP errors upon boot
RPCS3.log
But Oblivion seems to work now
RPCS3.log.gz

[kd-11]

Remaining crashes should be fixed now. I'll fix the non-windows builds later.

[Darkhost1999]

Remaining crashes should be fixed now. I'll fix the non-windows builds later.

All good now I'll keep testing more stuff

[ejams1]

I tested Killzone 2, I played a skirmish match and everything was stable as on the current build.

[digant73]

I also made some testing with my usual games (KZ2, KZ2, R1, R2, R3 etc.). With latest fixes it is ok

[ghost]

@kd-11 Hi, sorry for offtopic, please write me an e-mail if necessary, I don't know how to open Discord at the moment.

[kd-11]

@kd-11 Hi, sorry for offtopic, please write me an e-mail if necessary, I don't know how to open Discord at the moment.

No problem.

[kd-11]

Final round of testing needed. The shader cache version has been upped, we're tracking some new variables that I need for some future optimization work.

[LucioXerus]

I retested call of duty, need for speed, goldeneye and tekken using the linux gcc build. All games work properly as before. My audio issues had to do with cubeb and the audio format settings in the emulator so should be good to go.

[digant73]

I will retest this evening at home

[digant73]

tested latest changes with no issues. I hoped that the commit fixing the crash due to invalid buffer could also be related to the device-lost issue but unfortunately this was not the case