TextureConversionShader: Use integer math for truncating EFB format

[stenzek]

In relation to the Metroid Prime 2 regression from #5202, I'm guessing if the game is scanning by rendering fixed colors to the framebuffer, this may have been the cause of the issue. We do truncate/dither in the fragment shader already, so the conversion here may not be necessary anyway.

It would be great if someone with the game could test this build.

The behavior of the alpha channel remains the same, which should mean the original issue that #5202 tried to solve should still be fine.

Edit: I'm trying a version that uses integer math instead of removing the truncation entirely.

[JosJuice]

Tested and working: https://forums.dolphin-emu.org/Thread-metroid-prime-scanning-is-not-working-properly?pid=441935#pid441935

[MayImilae]

Using Metroid Prime 2 and 3, I reproduced the bug in 5.0-3463. Prime 2 was totally fine with the fix in this PR, but there's stiiilll a bit of a problem in Prime 3. Unlike before where it was always wrong, it is usually right, but if I aim at a specific part of the ship and scan it, it flips around and scans something else. This is hard to describe with text so I recorded a video.

....ok that didn't work! Dumping frames is doing something bananas right now. So um, I'm just going to describe it! In metroid prime 3 with this PR, if you target something directly, and see it light up and everything, then scan it, it will work fine. But if you try to scan something a liiitttle off, like not centered or a tiny bit away from the object, instead of not scanning or scanning the nearest object, it flips you around and scans an entirely different object somewhere else. This is a marked improvement from master, but, not correct at all. :/

Oh, note that Samus's ship is a great way to reproduce this bug, since it has a big open area underneath it while it's landed. It's how I caught it!

Btw, I only tested in OpenGL, for whatever that is worth.

Tested on:
Intel Core i5-3570k @ 4.6ghz
Nvidia GeForce GTX 1080
Windows 7 x64

[JMC47]

Bug Report says that Metroid Prime 3 is fixed in Vulkan in this pull request.

[MayImilae]

@JMC47 I really should check the other backends...

[0x647262]

This has fixed my scan-visor issues on OGL and Vulkan on Linux.

Driver: Mesa (amdgpu / radv)

[MayImilae]

Just retested on OGL, Vulkan, and D3D - everything is all fixed! LGTM!