Float64 rounding

[lrweck]

What version of Go are you using (go version)?

$ go version
go version go1.16.6 linux/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/luisweck/.cache/go-build"
GOENV="/home/luisweck/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/luisweck/Projects/Go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/luisweck/Projects/Go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/luisweck/Projects/Go/src/github.com/lrweck/gobserver/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2331568520=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Float64 rounding error
https://play.golang.org/p/zsevzyPMIu4

What did you expect to see?

1. 10000000000000000.00 - 339.16 = 9999999999999660.88
2. 10000000000000000.00 - 339.16 = 9999999999999660.88

What did you see instead?

if v has 16 places, Go just gives wrong answer:
1000000000000000.00 - 339.16 = 999999999999660.88

with 15 places, it does the right thing.

with 17, it eats up the decimal places:

1. 10000000000000000.00 - 339.16 = 9999999999999660.00
2. 10000000000000000.00 - 339.16 = 9999999999999660.00

[robpike]

This is how IEEE 754 works. There are 53 bits of mantissa in a float64, and 2**53 is 9007199254740992, which is 16 decimal digits. Your calculations run out of precision.

Please see https://en.wikipedia.org/wiki/Double-precision_floating-point_format, or for a fuller description, https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

[Fuerback]

This is how IEEE 754 works. There are 53 bits of mantissa in a float64, and 2**53 is 9007199254740992, which is 16 decimal digits. Your calculations run out of precision.

Please see https://en.wikipedia.org/wiki/Double-precision_floating-point_format, or for a fuller description, https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

1000000000000000.00 - 0.16 = 999999999999999.88

The example above there are 16 decimal digits, but the result is incorrect, would've to be 999999999999999.84

But with 15 decimal digits the result is correct:
100000000000000.00 - 0.16 = 99999999999999.84

[randall77]

This is again, how float64 works. You're running up against precision limits.

This program may help you see the representable values near where you are working.

package main

import (
	"fmt"
	"math"
)

func main() {
	v := float64(10000000000000000.00)
	for i := 0; i < 200; i++ {
		fmt.Printf("%.2f\n", v)
		v = math.Nextafter(v, 0)
	}
}

[thanm]

Note that Go provides packages such as https://pkg.go.dev/math/big for performing arbitrary-precision arithmetic.