rsx: Performance improvements and bug fixes

[kd-11]

Rewrites vertex upload, removing the slower methods used before. The old code is improved as well and kept around for debugging until its state is known to be mature enough.
Also lifts some restrictions on vertex register formats and adds a fix for register type data uploads when using a vertex buffer stream.
Overall, performance is up anywhere from 10-300% for geometry heavy scenes.

[SakataGintokiYT]

Vulkan:

Naruto: Ultimate Ninja Storm

E {rsx::thread} RSX: Warning, version 420 is not yet complete; most version-specific features are present, but some are missing.
ERROR: 0:151: '' : boolean expression expected 
ERROR: 1 compilation errors.  No code generated.


E {rsx::thread} RSX: 
F {rsx::thread} class std::runtime_error thrown: Failed to compile fragment shader
(in file C:\rpcs3\rpcs3\Emu\RSX\VK\VKFragmentProgram.cpp:476)

Log:

---

OpenGL:

Naruto: Ultimate Ninja Storm

E {rsx::thread} RSX: Failed to compile shader: 0(144) : error C1019: scalar Boolean expression expected

F {rsx::thread} class gl::glsl::link_exception thrown: linkage failed: 'Fragment info
-------------
0(144) : error C1019: scalar Boolean expression expected
(0) : error C2003: incompatible options for link
'

Log:

---

DX12:

Naruto: Ultimate Ninja Storm

E {rsx::thread} RSX: FS build failed:E:\Emulatory I Gry\PS3\rpcs3-v0.0.2-2017-04-30-28456170_win64\FragmentProgram.hlsl(228,7-40): error X3019: if statement conditional expressions must evaluate to a scalar
E:\Emulatory I Gry\PS3\rpcs3-v0.0.2-2017-04-30-28456170_win64\FragmentProgram.hlsl(229,2-3): error X3000: unrecognized identifier 'h3'

F {rsx::thread} class std::runtime_error thrown: fragment program compilation failure
(in file c:\rpcs3\rpcs3\emu\rsx\d3d12\D3D12PipelineState.h:162)

Log:

[twdarkeh]

Akiba's Trip, Trails of Cold Steel, and Final Fantasy X HD no longer work either, same error as Naruto.

[OverlordCW]

Same with RDR

E {rsx::thread} RSX: Failed to compile shader: 0(55) : error C1019: scalar Boolean expression expected

F {rsx::thread} class gl::glsl::link_exception thrown: linkage failed: 'Fragment info
-------------
0(55) : error C1019: scalar Boolean expression expected
(0) : error C2003: incompatible options for link

RPCS3.zip

[kd-11]

Fixed now. Hopefully. If not, I'll have to fix the decompiler as a separate PR.

[SakataGintokiYT]

@kd-11
Now work :)

[kd-11]

I am aware of some regressions in rare cases but I think we can track those separately.

[danilaml]

Finally D3 became playable! Before it took ages even to get past the intro.