cmd/compile: mips/mips64 floating point negation does not handle NaN correctly

[ianlancetaylor]

As discussed on #56491 (comment), on MIPS the gc compiler uses neg.d (which it calls NEGD) to negate a floating-point value. However, that instruction does not actually negate a NaN. That suggests that we should not use neg.d for floating point negation, but should instead xor with the high bit.

CC @randall77 @golang/mips @golang/runtime

[randall77]

It's not entirely clear to me that negating a NaN should result in a negative NaN.

NaN means something went wrong when computing this number. Its sign is kind of meaningless.

Operations with a NaN argument tend to return that NaN argument unchanged (or one of the NaN arguments, if there are several, not sure how the choice is made). For instance, explain this program:

-x for floats is clearly special on most archs, as it actually computes on the NaN instead of returning it unchanged. Except on mips, which to my mind is more consistent because it returns its NaN arg, even when the operation is negation.

Does ieee754 say anything about this?

[smasher164]

Does ieee754 say anything about this?

6.3 The sign bit
When either an input or result is a NaN, this standard does not interpret the sign of a NaN. However,
operations on bit strings—copy, negate, abs, copySign—specify the sign bit of a NaN result, sometimes
based upon the sign bit of a NaN operand. The logical predicates totalOrder and isSignMinus are also
affected by the sign bit of a NaN operand. For all other operations, this standard does not specify the sign
bit of a NaN result, even when there is only one input NaN, or when the NaN is produced from an invalid
operation.

---

It's unclear to me what "sometimes based upon the sign bit of a NaN operand" means here. Does that mean negate takes into account the sign bit or not?

[smasher164]

Okay from section 5.5.1:

For x a NaN, the results of the operations have the same representation as x (qNaN or sNaN).

The operations it's referring to are copy, negate, abs, and copySign. So I guess the sign for NaN is undefined.

[randall77]

Does that mean (-1) * x and -x have different semantics? Ugh. We do that optimization in the generic rewrite rules.

Indeed: https://go.dev/play/p/W2R_eSU1X9G

[ianlancetaylor]

I don't think that the multiplication and the negation have to be implemented differently. IEEE doesn't specify the sign bit of a NaN result of multiplication, so it's OK to treat it as negation.