Jit64: Turn SNaN into QNaN in HandleNaNs

[JosJuice]

Improves accuracy but isn't known to affect any games.

This turned out to be fairly convenient to implement; ORing with the PPC default NaN will quieten SNaNs and do nothing to QNaNs.

[AdmiralCurtiss]

I don't know whether/why this is more accurate, but the non-paired code looks correct to me if your intent is to turn signaling NaNs into quiet NaNs by forcing bit 51 to 1.

The paired code I'm not sure I fully understand.

    avx_op(&XEmitter::VCMPPD, &XEmitter::CMPPD, clobber, R(xmm), R(xmm), CMP_UNORD);
    ANDPD(clobber, MConst(psGeneratedQNaN));
    ORPD(xmm, R(clobber));

If I pseudocode this out, this essentially is:

if (isnan(xmm)) {
  clobber = /* all bits 1 */;
} else {
  clobber = /* all bits 0 */;
}
clobber &= /* quiet nan */;
xmm |= clobber;

Or in other words,

if (isnan(xmm)) {
  xmm |= /* quiet nan */;
} else {
  // no change to xmm
}

Ah, okay, yes that actually makes sense!

[AdmiralCurtiss]

Code-wise approval, dunno if this is actually more accurate or not.

[JMC47]

have we checked to see if this fixes desyncs in 1080 Avalanche and friends with real console replays?

[JosJuice]

@AdmiralCurtiss dolphin-emu/hwtests#54 showed that this is what a real Wii does. Supposedly this behavior is specified in IEEE 754, but I can't find anywhere where it actually says that.

@JMC47 No. I believe you tested AccurateNaNs before and it didn't seem to have any impact, in which case it's unlikely that this PR would have any impact (with AccurateNaNs enabled). But feel free to test it if you want to.