HLE RSP produces inconsistent dumps

[GhostlyDark]

Dumping JPEG backgrounds with GLideN64 gives different results per RSP plugin. So all the backgrounds of Ocarina of Time have different hashes between cxd4, PJ64 RSP and the mupen64plus HLE plugin. That means 36 background dumps times 3, which I assume happens due to differences in how the RSP plugins decode those JPEG images.

This isn't much of an issue. Just dump all of those duplicates and retexture them, one can live with that. However, with the mupen64plus HLE RSP, the plugin may produce even more different results. It's not a new set of 36 new dumps, but instead it can range from 3 to 10 or so different dumps.

It seems that depending on the targeted architecture, the compiler used and depending on what optimization flags have been chosen, certain backgrounds are dumped with a different hash. They look visually identical and tend to be the same size, but obviously aren't the same files, as they differ with a few bytes of differences.

Targeting x64 and using either gcc or clang seems to be consistent. I use that as the "correct" reference.
Targeting either x64 or x86 using Visual Studio results in a couple of new dumps.
Targeting x86 using gcc or clang gives a number of different results, depending on what compiler and what optimization flags have been used.

I expect the RSP plugin to always decode the JPEG images the same regardless of how the RSP was compiled, limiting the outcome of the HLE plugin to only 36 dumps.

I haven't tested any other games that use JPEG backgrounds aside from Resident Evil 2, which shows no such issue. (Side note: RE2 backgrounds dumped while using the m64p HLE vs. cxd4/pj64 RSP look visually different and differ in file size significantly. But that's another issue.)

As an example, the Market Main Alley background:

x64 gcc/clang:
THE LEGEND OF ZELDA#4D4BFEDD#0#2_all.png

Visual Studio x86/x64:
THE LEGEND OF ZELDA#A6BEC658#0#2_all.png

gcc/clang x86:
THE LEGEND OF ZELDA#8F090C10#0#2_all.png
THE LEGEND OF ZELDA#EFCC243D#0#2_all.png

ZIP file with all dumps, including other RSPs and some of those duplicates that originate from different compiler settings:
oot-backgrounds.zip

[bsmiles32]

Current implementation of JPEG decoding task in RSP HLE are not bit-exact as you've discovered. The implementation uses float internally and due to how compilers are allowed to optimize such instructions different compilers may generate different results. This is something that we cannot fix easily by just switching some compiler flags (or I don't know how that's possible).
Internally the RSP does computations using fixed-point arithmetics, so if we want to match exactly what the hardware produces (and not just something visually close) we should reimplement the JPEG ucode with fixed-point arithmetics like the RSP does.

Another possibility (but I don't really like that option) is to disable JPEG decoding in RSP HLE (by commenting https://github.com/mupen64plus/mupen64plus-rsp-hle/blob/master/src/hle.c#L343-L355) and enable task forwarding to cxd4 RSP via RspFallback option. That way you bypass HLE implementation of JPEG decoding and handle it through cxd4 RSP which gives more reproducible results.