math: math.SmallestNonzeroFloat64 is not sufficiently precise, leading to unexpected results

[griesemer]

The compiler constant expression

math.SmallestNonzeroFloat64 / 2

gets printed as 5e-324 (program), but the exact computation of math.SmallestNonzeroFloat64 / 2 using math/big arithmetic followed by rounding results in 0 (program).

There's either an error with the rounding or constant arithmetic somewhere, or a mistake in my assumptions.

This affects a test case in types2/testdata/const1.src.

[dreamerjackson]

i think it's a mistake, in golang compile, the math.SmallestNonzeroFloat64 / 2 will use big.Float to calculate multi-precision floating， and when transform to float64 the precision will loss, so the print will undefined.

and your example use big.Rat, it's different

[dreamerjackson]

math.SmallestNonzeroFloat64 / 2 is a special example will rounding to math.SmallestNonzeroFloat64, the logic is here。change the types2/testdata/const1.src

func (x *Float) Float64() (float64, Accuracy) {
		if e < emin {
			// recompute precision
			p = mbits + 1 - emin + int(e)
			if p < 0 /* m <= 0.25 */ || p == 0 && x.mant.sticky(uint(len(x.mant))*_W-1) == 0 /* m == 0.5 */ {
				// underflow to ±0
				if x.neg {
					var z float64
					return -z, Above
				}
				return 0.0, Below
			}
			// otherwise, round up
			// We handle p == 0 explicitly because it's easy and because
			// Float.round doesn't support rounding to 0 bits of precision.
			if p == 0 {
				if x.neg {
					return -math.SmallestNonzeroFloat64, Below
				}
				return math.SmallestNonzeroFloat64, Above
			}
		}
}

[gopherbot]

Change https://golang.org/cl/288672 mentions this issue: math/big: fix SmallestNonzeroFloat64 const tesedata data

[griesemer]

math.SmallestNonzeroFloat64 has the value 4.940656458412465441765687928682213723651e-324. This appears to not be sufficiently precise a representation (it's rounded up and a bit too large). If we add 3 more digits and go to 4.940656458412465441765687928682213723650598e-324 then dividing this number by 2 results in a float64 value that is 0, as expected (program).

[griesemer]

Program to compute accurate values to 512 bits.

[gopherbot]

Change https://golang.org/cl/315170 mentions this issue: math: increase precision of math.SmallestNonzeroFloat64

[gopherbot]

Change https://golang.org/cl/315969 mentions this issue: math: replace float32/64 extrema with exact expressions