math: Max(NaN, x) may return x instead of NaN (depending on the bit representation of NaN)

[barnex]

The following code: http://play.golang.org/p/uvj_pmygbE prints math.Max(NaN, 0), where NaN is obtained by either math.NaN() or a/a with a := 0.0.

In the second case, the result is 0 instead of the expected NaN.

This issue is shared by math.Min. However, math.IsNaN works fine for both NaN's.

Tested with go 1.4.2 on linux amd64 and with go 1.4.1 in the playground on nacl amd64p32.

[barnex]

The issue can be traced back to math/dim_amd64.s. Here, checking for NaN is done by comparing the number with 0x7FF8000000000001. Instead, we need a check that works for all NaN bit patterns.

[cldorian]

The bit representation of NaN was changed (https://go.googlesource.com/go/+/6e9506a7b45958665c3f48deecc8555f3ee2c42b%5E!/src/pkg/math/bits.go) to match gcc for a reason. I think the problem may be that 0/0 is generating the wrong bit pattern for NaN. That said, changing the function to do a mask and compare for NaN would be more general. I'll take a look.

[cldorian]

The bit pattern starting 0xFFF8000... comes from the x64 DIVSD instruction. I've submitted change 5553.