GSDX user hack for incorrect sprite UV rounding

[hibye8313]

GSDX user hack for incorrect sprite UV rounding in game Cho Aniki: Seinaru Protein Densetsu (and possibly others?).

This PR is in reference to the sprite issues at #3552. The problem was that sprites were being drawn whose UV coords at the right-most/bottom-most pixels were exactly on the boundary of 2 texels. The texel on the top/left of the boundary was the correct one but it seems OpenGl, DirectX, and the SW renderer all round up (to the bottom/right) resulting in incorrect texels. The hack here is basically to detect cases where:

1. We are drawing a textured sprite with UV coords
2. The UV coords (after interpolation) of the right-most/bottom-most pixels are exactly on a texel boundary (happens when the interpolated U or V is an exact integer).
   If this happens we just subtract 1/16 from the input U (or V) so that they will be rounded down instead of up in the final texturing.

So far this is only tested to work on a single GS dump of a single game referenced above on the issue. If anyone else knows of similar issues that I can test out (or even better has the dump) it would be greatly appreciated.

This hack affect both the HW renderer and the SW renderer because it was inserted far enough up the pipeline. There might be a more accurate fix for just the SW renderer but TBH I haven't messed much with the SW renderer and I don't know the true behavior of the GS is to know what is the most accurate way to do things (can anyone reading this run tests on an actual PS2?).

In the GUI I had to put this under "HW Hacks" because there is nowhere else for hacks that affect both HW/SW. If anyone thinks this should only be a HW hack only please let me know.

There are a few other hacks that deal with sprite offset issues but it seems as if they are only meant for upscaling issues (I also tried to force enable them for native res and they did not fix this issue). The code for this is actually based on another upscaling hack (RoundSpriteOffset) but they do slightly different things (in particular this hack only affects the bottom/right boundaries of the sprite instead of all boundaries).

Any comments/criticism would be very much welcome and appreciated. Thanks!

[lightningterror]

I'm not really a fan of adding hacks, especially on sw renderer side, it's better to find a solution that properly works.

[hibye8313]

@lightningterror I agree it's better not to add hacks. My hope is that this behavior does not break anything and only improves accuracy with games that need it. If this is true then it will hopefully be more of a "workaround" than a hack since otherwise OGL and DX might be unable to accurately emulate the correct behavior without some kind of finagling. I left it as a hack because of the possible perf impact and because it needs more research with other games with similar issues (as you pointed out in the tag). Would this PR be more acceptable if I only put it in the HW renderer? It is a very small change to make.

I can try and dig into the SW renderer and see if I can figure out how to fix the issue properly. IDK maybe there are more important issues to work on right now than this one.

[RedDevilus]

It would be best as a real fix or as a workaround (patch) for gameindex.dbf file. Maybe look into researching if it's possible.
Reason:
#3568

[refractionpcsx2]

@tadanokojin is already looking in to this to see if he can work out a proper fix, we really don't really want to be adding hacks to the software renderer, but there's a few games that suffer from this issue (even on Sony's emulator), but hopefully he can come up with a good solution

[hibye8313]

@RedDevilus Ok, that seems like a good way to handle the issue. I'll try and see If I can figure out a way to fix the issue for just that one game using a patch and close this PR.

@refractionpcsx2 Sounds good, if there is an actual fix for the issue it would be great. I have a couple ideas also that may be a way of fixing the issue such as breaking the sprite and rendering the bottom/right pixels separate from the rest or putting in some kind of special handling in the pixel shader that will adjust the the UV coords based on whether the pixel is a bottom/right pixel. I have a feeling that either of these ways might have a perf impact but I can try.

I really wish I could runs tests on an actual PS2, I don't understand why the GS would rasterize the bottom/right of the sprite differently from the rest.