Repository navigation
compress/flate: EstimatedBits output depends on CPU architecture ["fused multiply and add" (FMA)] #81589
Description
Go version
go version go1.27.1 darwin/arm64
Output of go env in your module/workspace:
AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/malte/go/bin'
GOCACHE='/Users/malte/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/malte/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/hz/zbtrlcls4257v1kdjg0qtnt00000gn/T/go-build540391637=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/malte/github.com/malt3/pgzip-detect/go.mod'
GOMODCACHE='/Users/malte/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPACKAGESDRIVER=''
GOPATH='/Users/malte/go'
GOPRIVATE=''
GOPROXY='REDACTED|direct'
GOROOT='/opt/homebrew/Cellar/go/1.27.1/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/malte/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.27.1/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.27.1'
GOWORK=''
PKG_CONFIG='pkg-config'What did you do?
I'm using compress/gzip to compress data.
I created an in-depth reproducer using an example payload that has a different compressed stream on arm64 vs amd64: https://github.com/malt3/pgzip-cpu-dependent-output-repro
The root cause is with the EstimatedBits and mFastLog2 functions in compress/flate/token.go. For example:
log2 += ((-0.34484843)*uval+2.02466578)*uval - 0.67487759
https://godbolt.org/z/dTzxMb8a6
The Go spec allows this to be replaced with a "fused multiply and add" (FMA), which changes how the result is rounded. FMA is used with GOARCH=aarch64, but not with GOARCH=amd64 GOAMD64=v1.
What did you see happen?
I used the test data from https://github.com/malt3/pgzip-cpu-dependent-output-repro and compressed it with compress/flate (as well as klauspost/pgzip and klauspost/compress). The exact code can be found in the reproducer.
What did you expect to see?
I understand that the behaviour (and expected output) of the compression changed with Go 1.27 (https://go-review.googlesource.com/c/go/+/707355) and users shouldn't depend on the exact bytes of the output.
However, I would expect that the same version of Go has a deterministic implementation of gzip (even across CPU architectures).
This is not true in my tests.
Activity
gabyhelp commentedon Sep 18, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
compress/flate: avoid FMA in mFastLog2 for portable rounding
gopherbot commentedon Sep 18, 2026
Change https://go.dev/cl/834885 mentions this issue:
compress/flate: avoid FMA in mFastLog2 for portable rounding[-]compress/flate: mFastLog2 output depends on CPU architecture ["fused multiply and add" (FMA)][/-][+]compress/flate: EstimatedBits output depends on CPU architecture ["fused multiply and add" (FMA)][/+]flate: avoid FMA in EstimatedBits for portable rounding
jakebailey commentedon Sep 18, 2026
I think this is actually pretty critical to fix, because the linker does DWARF compression and therefore binaries may not be reproducible depending on the FMA behavior of the system.
(Generally to me this means that implicit FMA should be outright banned in cmd/compile and cmd/link, maybe more, but that's a more drastic opinion.)
thepudds commentedon Sep 18, 2026
This might also be affected by
GOAMD64=v1versusGOAMD64=v3?malt3 commentedon Sep 18, 2026
@thepudds yes. See the godbolt link above. It shows that the issue also exists on GOAMD64=v3: https://godbolt.org/z/dTzxMb8a6
cherrymui commentedon Sep 18, 2026
cc @klauspost @dsnet
flate: avoid FMA in EstimatedBits for portable rounding
flate: avoid FMA in EstimatedBits for portable rounding
malt3 commentedon Sep 30, 2026
Gentle ping on this. The same fix was already merged in klauspost/compress.
Given the statement from @jakebailey above, I think we may even want to backport this to Go 1.27:
jakebailey commentedon Sep 30, 2026
I might be wrong about it; when I looked and tried to reproduce a bug, the FMA code wasn't actually hit because the DWARF compression level used by the linker is not actually high enough. Still, bad bug!