PixelShaderGen: Manually wrap negative indirect texture coordinates

[stenzek]

(x % y) is not defined in GLSL when sign(x) != sign(y).
This also has the added benefit of behaving the same as sampler wrapping modes, in regards to negative inputs.

This was causing raindrops to render incorrectly in F-Zero GX with OGL on windows/intel and mesa/llvmpipe, as well as D3D, due to the previous implementation invoking UB when the coordinates were negative. (see GLSL spec re % operator, D3D seems to take the sign from the LHS)

Marked WIP because we're not sure if this actually matches hardware behavior in regards to how negative coordinates are handled.

[delroth]

@JMC47 is this a candidate for 5.0?

[degasus]

It fixes an undefined behavior bug, but it may lead to unknown regressions ...

[JMC47]

@delroth imo yes, as it brings D3D/OpenGL even on a few graphical bugs. It also makes it a lot easier for me to cleanup the wiki on those two bugs.

[BhaaLseN]

Those two diffs look good to me tho. Did you try writing a HW-Test yet?

[degasus]

As all of our guesses which returns negative values didn't fix those two fifologs, I think this is the only possible way for the hardware. LGTM when the coding style issues are fixed.

[dolphin-emu-bot]

FifoCI detected that this change impacts graphical rendering. Here are the behavior differences detected by the system:

• f-zero-rain on ogl-lin-mesa: diff
• ss-map on ogl-lin-mesa: diff
• f-zero-rain on sw-lin-mesa: diff
• ss-map on sw-lin-mesa: diff

_{^{automated-fifoci-reporter}}

[JMC47]

This probably isn't important to merge due to the bug being graphical, but at the same time it'd make F-Zero GX's updated wiki page a lot cleaner if we were able to outright remove the bug instead of noting it for another release cycle. I know that's stupid; but in TFN, that was one of the issues we advertised as fixing