RSX: Use checksum with rotated inputs for shader hashing instead of FNV

[Whatcookie]

Raises FPS from ~180 to ~200 in Ninja Gaiden Sigma

It's difficult to verify how good the hashing is myself, so please test with a lot of games, checking for any visual issues and performance improvements. I do think that simpler hashes should be sufficient when we are hashing valid opcodes instead of random data though.

Before:

After:

Note: The functions are small enough now that the compiler decided to inline them. In the before screenshot get_vertex_program_ucode_hash is using 1.65% of cpu time. In the after screenshot, the function is inlined into vertex_program_storage_hash and the combined code is only taking up 0.56% of cpu time.

NoteNote: I tested without the rotation, and none of my games had issues, but I was nervous about instructions getting swapped around, so I added the rotation step.

NoteNoteNote: This code could be trivially vectorized with AVX10 or AVX-512, with masked loads and rotates. (AVX2 doesn't have rotates!)

[kd-11]

What does this look like on arm64? Use godbolt to verify it doesn't spit out emulated stuff. Min target is cortex-a78.
Though looking at the original code, it could have been vectorized to do the shift and horizontal sum.
Now, with shader hashing the side-effect won't likely be glitchy visuals. The hash key has a comparator doing bitwise comparisons for each entry. Instead you'll probably get collisions on store, causing "compiling shaders" step to happen even when the user has been through an area. That is going to be the real test.

[Whatcookie]

What does this look like on arm64? Use godbolt to verify it doesn't spit out emulated stuff. Min target is cortex-a78.

Codegen on x86 and a78 are nearly identical:

Though looking at the original code, it could have been vectorized to do the shift and horizontal sum.

Yeah I guess, though you'd have to do a horizontal sum for each loop iteration, I can't imagine it'd be very fast.

[Whatcookie]

I guess I can use rotr instead, since the a78 seems to be emulating rotl with an inverted rotate count, and save 1 instruction for arm.

[Whatcookie]

Pushed another commit with another 1% performance uplift:
In fragment_program_compare we checked both const bool end = ((inst1._u32[0] >> 8) & 0x1) && ((inst2._u32[0] >> 8) & 0x1);, despite having earlier in the loop confirmed that both data streams are identical. We can just check one stream for the end.

In vertex _program_compare we checked if both data streams have the same instruction mask. But if the instruction masks differed, it would've appeared in the previous iteration of the loop, where the instructions would've compared differently. So skip comparing the instruction masks.

Also added a silly microptimization with is_pair_constant that saves 1 instruction per loop.

[elad335]

Probably better to test in production here.