Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: ethereum build not reproducible #59500

Closed
jrick opened this issue Apr 8, 2023 · 73 comments
Closed

cmd/go: ethereum build not reproducible #59500

jrick opened this issue Apr 8, 2023 · 73 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@jrick
Copy link
Contributor

jrick commented Apr 8, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20.3 windows/amd64

$ go version
go version go1.20.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env (windows) Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\jrick\AppData\Local\go-build
set GOENV=C:\Users\jrick\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\jrick\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\jrick\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.3
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\jrick\src\go-ethereum\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\jrick\AppData\Local\Temp\go-build1436174171=/tmp/go-build -gno-record-gcc-switches
go env (linux) Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jrick/.cache/go-build"
GOENV="/home/jrick/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jrick/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jrick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib64/go/1.20"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib64/go/1.20/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jrick/src/go-ethereum/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build632484733=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Performing a cross compile, that should be bit-for-bit reproducible, of the geth executable from the go-ethereum repo checked out at commit b946b7a13b749c99979e312c83dce34cac8dd7b1 results in different file hashes when building on Linux vs Windows.

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 windows/amd64
-rwxr-xr-x 1 jrick 197609 40166400 Apr  8 08:32 ./geth.exe*
SHA256(geth.exe)= 6d065b5978219c934d688e952fd658a58538d324c0e206f0c46f253e43b5a39a

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 linux/amd64
-rwxr-xr-x 1 jrick jrick 40152064 Apr  8 12:31 geth.exe
SHA2-256(geth.exe)= 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75

What did you expect to see?

Same executable built form both hosts. These are the same procedures we use for reproducible cross compiles of other packages, and have not had an issue until building this project.

What did you see instead?

Differing builds

@jrick jrick changed the title cmd/go: build of github.com/ethereum/go-ethereum/cmd/geth is not reproducible between Linux and Windows cmd/go: build of github.com/ethereum/go-ethereum/cmd/geth is not reproducible between Linux and Windows build hosts Apr 8, 2023
@jrick
Copy link
Contributor Author

jrick commented Apr 8, 2023

Some additional info: I can reproduce the same build as done by Linux from OpenBSD. The Windows build is the odd one out.

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe 
go version go1.20.3 openbsd/amd64
-rwxr-xr-x  1 jrick  jrick  40152064 Apr  8 08:55 geth.exe*
SHA256(geth.exe)= 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75

@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 8, 2023
@seankhliao
Copy link
Member

can you get the go version -m $binary output of the built artifacts?

@jrick
Copy link
Contributor Author

jrick commented Apr 8, 2023

Sure. The output is the same for both.

./geth.exe: go1.20.3
        path    github.com/ethereum/go-ethereum/cmd/geth
        mod     github.com/ethereum/go-ethereum (devel)
        dep     github.com/StackExchange/wmi    v0.0.0-20180116203802-5d049714c4a6      h1:fLjPD/aNc3UIOA6tDi6QXUemppXK3P9BI7mr2hd6gx8=
        dep     github.com/VictoriaMetrics/fastcache    v1.6.0  h1:C/3Oi3EiBCqufydp1neRZkqcwmEiuRT9c3fqvvgKm5o=
        dep     github.com/beorn7/perks v1.0.1  h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
        dep     github.com/btcsuite/btcd/btcec/v2       v2.2.0  h1:fzn1qaOt32TuLjFlkzYSsBC35Q3KUjT1SwPxiMSCF5k=
        dep     github.com/cespare/xxhash/v2    v2.2.0  h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
        dep     github.com/cockroachdb/errors   v1.9.1  h1:yFVvsI0VxmRShfawbt/laCIDy/mtTqqnvoNgiy5bEV8=
        dep     github.com/cockroachdb/logtags  v0.0.0-20230118201751-21c54148d20b      h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE=
        dep     github.com/cockroachdb/pebble   v0.0.0-20230209160836-829675f94811      h1:ytcWPaNPhNoGMWEhDvS3zToKcDpRsLuRolQJBVGdozk=
        dep     github.com/cockroachdb/redact   v1.1.3  h1:AKZds10rFSIj7qADf0g46UixK8NNLwWTNdCIGS5wfSQ=
        dep     github.com/cpuguy83/go-md2man/v2        v2.0.2  h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
        dep     github.com/crate-crypto/go-ipa  v0.0.0-20220523130400-f11357ae11c7      h1:6IrxszG5G+O7zhtkWxq6+unVvnrm1fqV2Pe+T95DUzw=
        dep     github.com/davecgh/go-spew      v1.1.1  h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
        dep     github.com/deckarep/golang-set/v2       v2.1.0  h1:g47V4Or+DUdzbs8FxCCmgb6VYd+ptPAngjM6dtGktsI=
        dep     github.com/decred/dcrd/dcrec/secp256k1/v4       v4.0.1  h1:YLtO71vCjJRCBcrPMtQ9nqBsqpA1m5sE92cU+pd5Mcc=
        dep     github.com/deepmap/oapi-codegen v1.8.2  h1:SegyeYGcdi0jLLrpbCMoJxnUUn8GBXHsvr4rbzjuhfU=
        dep     github.com/dlclark/regexp2      v1.7.0  h1:7lJfhqlPssTb1WQx4yvTHN0uElPEv52sbaECrAQxjAo=
        dep     github.com/dop251/goja  v0.0.0-20230122112309-96b1610dd4f7      h1:kgvzE5wLsLa7XKfV85VZl40QXaMCaeFtHpPwJ8fhotY=
        dep     github.com/edsrzf/mmap-go       v1.0.0  h1:CEBF7HpRnUCSJgGUb5h1Gm7e3VkmVDrR8lvWVLtrOFw=
        dep     github.com/fatih/color  v1.7.0  h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
        dep     github.com/fjl/memsize  v0.0.0-20190710130421-bcb5799ab5e5      h1:FtmdgXiUlNeRsoNMFlKLDt+S+6hbjVMEW6RGQ7aUf7c=
        dep     github.com/gballet/go-libpcsclite       v0.0.0-20190607065134-2772fd86a8ff      h1:tY80oXqGNY4FhTFhk+o9oFHGINQ/+vhlm8HFzi6znCI=
        dep     github.com/gballet/go-verkle    v0.0.0-20220902153445-097bd83b7732      h1:AB7YjNrzlVHsYz06zCULVV2zYCEft82P86dSmtwxKL0=
        dep     github.com/getsentry/sentry-go  v0.18.0 h1:MtBW5H9QgdcJabtZcuJG80BMOwaBpkRDZkxRkNC1sN0=
        dep     github.com/go-ole/go-ole        v1.2.1  h1:2lOsA72HgjxAuMlKpFiCbHTvu44PIVkZ5hqm3RSdI/E=
        dep     github.com/go-sourcemap/sourcemap       v2.1.3+incompatible     h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
        dep     github.com/go-stack/stack       v1.8.1  h1:ntEHSVwIt7PNXNpgPmVfMrNhLtgjlmnZha2kOpuRiDw=
        dep     github.com/gofrs/flock  v0.8.1  h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw=
        dep     github.com/gogo/protobuf        v1.3.2  h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
        dep     github.com/golang-jwt/jwt/v4    v4.3.0  h1:kHL1vqdqWNfATmA0FNMdmZNMyZI1U6O31X4rlIPoBog=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/golang/snappy        v0.0.5-0.20220116011046-fa5810519dcb    h1:PBC98N2aIaM3XXiurYmW7fx4GZkL8feAMVq7nEjURHk=
        dep     github.com/google/uuid  v1.3.0  h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
        dep     github.com/gorilla/websocket    v1.4.2  h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
        dep     github.com/graph-gophers/graphql-go     v1.3.0  h1:Eb9x/q6MFpCLz7jBCiP/WTxjSDrYLR1QY41SORZyNJ0=
        dep     github.com/hashicorp/go-bexpr   v0.1.10 h1:9kuI5PFotCboP3dkDYFr/wi0gg0QVbSNz5oFRpxn4uE=
        dep     github.com/holiman/bloomfilter/v2       v2.0.3  h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao=
        dep     github.com/holiman/uint256      v1.2.2-0.20230321075855-87b91420868c    h1:DZfsyhDK1hnSS5lH8l+JggqzEleHteTYfutAiVlSUM8=
        dep     github.com/huin/goupnp  v1.0.3  h1:N8No57ls+MnjlB+JPiCVSOyy/ot7MJTqlo7rn+NYSqQ=
        dep     github.com/influxdata/influxdb-client-go/v2     v2.4.0  h1:HGBfZYStlx3Kqvsv1h2pJixbCl/jhnFtxpKFAv9Tu5k=
        dep     github.com/influxdata/influxdb1-client  v0.0.0-20220302092344-a9ab5670611c      h1:qSHzRbhzK8RdXOsAdfDgO49TtqC1oZ+acxPrkfTxcCs=
        dep     github.com/influxdata/line-protocol     v0.0.0-20210311194329-9aa0e372d097      h1:vilfsDSy7TDxedi9gyBkMvAirat/oRcL0lFdJBf6tdM=
        dep     github.com/jackpal/go-nat-pmp   v1.0.2  h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
        dep     github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e      h1:UvSe12bq+Uj2hWd8aOlwPmoZ+CITRFrdit+sDGfAg8U=
        dep     github.com/karalabe/usb v0.0.2  h1:M6QQBNxF+CQ8OFvxrT90BA0qBOXymndZnk5q235mFc4=
        dep     github.com/klauspost/compress   v1.15.15        h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw=
        dep     github.com/kr/pretty    v0.3.1  h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
        dep     github.com/kr/text      v0.2.0  h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
        dep     github.com/mattn/go-colorable   v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
        dep     github.com/mattn/go-isatty      v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ=
        dep     github.com/mattn/go-runewidth   v0.0.9  h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
        dep     github.com/matttproud/golang_protobuf_extensions        v1.0.4  h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
        dep     github.com/mitchellh/mapstructure       v1.4.1  h1:CpVNEelQCZBooIPDn+AR3NpivK/TIKU8bDxdASFVQag=
        dep     github.com/mitchellh/pointerstructure   v1.2.0  h1:O+i9nHnXS3l/9Wu7r4NrEdwA2VFTicjUEN1uBnDo34A=
        dep     github.com/naoina/go-stringutil v0.1.0  h1:rCUeRUHjBjGTSHl0VC00jUPLz8/F9dDzYI70Hzifhks=
        dep     github.com/naoina/toml  v0.1.2-0.20170918210437-9fafd6967416    h1:shk/vn9oCoOTmwcouEdwIeOtOGA/ELRUw/GwvxwfT+0=
        dep     github.com/olekukonko/tablewriter       v0.0.5  h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
        dep     github.com/opentracing/opentracing-go   v1.1.0  h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU=
        dep     github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7    h1:oYW+YCJ1pachXTQmzR3rNLYGGz4g/UgFcjb28p/viDM=
        dep     github.com/pkg/errors   v0.9.1  h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
        dep     github.com/prometheus/client_golang     v1.14.0 h1:nJdhIvne2eSX/XRAFV9PcvFFRbrjbcTUj0VP62TMhnw=
        dep     github.com/prometheus/client_model      v0.3.0  h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4=
        dep     github.com/prometheus/common    v0.39.0 h1:oOyhkDq05hPZKItWVBkJ6g6AtGxi+fy7F4JvUV8uhsI=
        dep     github.com/rogpeppe/go-internal v1.9.0  h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
        dep     github.com/rs/cors      v1.7.0  h1:+88SsELBHx5r+hZ8TCkggzSstaWNbDvThkVK8H6f9ik=
        dep     github.com/russross/blackfriday/v2      v2.1.0  h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
        dep     github.com/shirou/gopsutil      v3.21.4-0.20210419000835-c7a38de76ee5+incompatible      h1:Bn1aCHHRnjv4Bl16T8rcaFjYSrGrIZvpiGO6P3Q4GpU=
        dep     github.com/status-im/keycard-go v0.2.0  h1:QDLFswOQu1r5jsycloeQh3bVU8n/NatHHaZobtDnDzA=
        dep     github.com/syndtr/goleveldb     v1.0.1-0.20210819022825-2ae1ddf74ef7    h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY=
        dep     github.com/tyler-smith/go-bip39 v1.1.0  h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8=
        dep     github.com/urfave/cli/v2        v2.17.2-0.20221006022127-8f469abc00aa   h1:5SqCsI/2Qya2bCzK15ozrqo2sZxkh0FHynJZOTVoV6Q=
        dep     github.com/xrash/smetrics       v0.0.0-20201216005158-039620a65673      h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=
        dep     golang.org/x/crypto     v0.1.0  h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU=
        dep     golang.org/x/exp        v0.0.0-20230206171751-46f607a40771      h1:xP7rWLUr1e1n2xkK5YB4LI0hPEy3LJC6Wk+D4pGlOJg=
        dep     golang.org/x/net        v0.8.0  h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
        dep     golang.org/x/sync       v0.1.0  h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
        dep     golang.org/x/sys        v0.6.0  h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
        dep     golang.org/x/text       v0.8.0  h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
        dep     golang.org/x/time       v0.0.0-20220922220347-f3bd1da661af      h1:Yx9k8YCG3dvF87UAn2tu2HQLf2dt/eR1bXxpLMWeH+Y=
        dep     google.golang.org/protobuf      v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
        dep     gopkg.in/natefinch/lumberjack.v2        v2.0.0  h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
        dep     gopkg.in/natefinch/npipe.v2     v2.0.0-20160621034901-c1b8fa8bdcce      h1:+JknDZhAj8YMt7GC73Ei8pv4MzjDUNPHgQWJdtMAaDU=
        dep     gopkg.in/yaml.v2        v2.4.0  h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

@bcmills bcmills added the GoCommand cmd/go label Apr 8, 2023
@bcmills bcmills added this to the Go1.21 milestone Apr 8, 2023
@jrick
Copy link
Contributor Author

jrick commented Apr 8, 2023

Here's an attempt at diffing the binaries. (I removed the -s -w linker flags for this)

$ go tool objdump geth.exe.linux | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > a
$ go tool objdump geth.exe.windows | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > b
$ diff -u a b

diff output: https://gist.github.com/jrick/c3b3b2ad54212a6a634d66997d4d47b3

@jrick
Copy link
Contributor Author

jrick commented Apr 8, 2023

The differences in the text compiled for element_mul_amd64.s seems to be the most likely cause?

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

I am able to reproduce the 6d065b5978219c934d688e952fd658a58538d324c0e206f0c46f253e43b5a39a checksum on a Windows host, and the 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75 checksum on a Linux host.

In order to rule out differences in git behavior between the two platforms, I also tried building geth.exe using go install. I was able to reproduce a difference there too: when built on linux I see

bryan@guthrie:~/tmp$ GOOS=windows GOARCH=amd64 GOAMD64=v1 CGO_ENABLED=0 GOFLAGS= GOWORK=off go1.20.3 install -buildvcs=false -trimpath -ldflags='-buildid= -s -w' github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74 && openssl sha256 $(go env GOPATH)/bin/windows_amd64/geth.exe
SHA256(/home/bryan/bin/windows_amd64/geth.exe)= 826ab71fc227add7d4403a85843e7e6ad6cddc80bdcfa844ba2af18fd9b3d2f6
C:\tmp>"C:\program files\go\bin\go" install -buildvcs=false -trimpath "-ldflags=-buildid= -s -w" github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74

C:\tmp>"C:\program files\go\bin\go" env GOPATH
C:\Users\bryan\go
bryan@guthrie:/mnt/c/tmp$ openssl sha256 /mnt/c/Users/bryan/go/bin/geth.exe
SHA256(/mnt/c/Users/bryan/go/bin/geth.exe)= 4dd9da4ce478efdedcf0ba42639b5b05d0385ab96d069b9f59a34946c8289775

go version -m confirms that they were build with identical dependency versions and build settings.

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

Curiously, building using a Go compiler in a WSL mount of the Windows clone of the repo produces yet a third checksum. 😵‍💫

bryan@guthrie:/mnt/c/Users/bryan/src/github.com/ethereum/go-ethereum$ go1.20.3 version && GOOS=windows GOARCH=amd64 GOAMD64=v1 CGO_ENABLED=0 GOFLAGS= GOWORK=off go1.20.3 build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 linux/amd64
-rwxrwxrwx 1 bryan bryan 40166400 Apr 10 11:05 geth.exe
SHA256(geth.exe)= 7875cb8cc4b26fd241f65f7f2ce59c4d9237fad4bd4c9cea029e43435ee701bb

@bcmills bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 10, 2023
@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

Output from go install -a -x -buildvcs=false -trimpath "-ldflags=-buildid= -s -w" github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74 on both Windows and Linux, targeting the same Windows configuration:
go-windows.log
go-cross.log

Those builds reproduced the 4dd9… checksum on Windows and the 826a… checksum on Linux respectively.

@golang/compiler: at first blush this looks like a compiler and/or linker reproducibility bug to me. Note that these are for pure-Go builds (with CGO_ENABLED=0).

@prattmic
Copy link
Member

I'm not 100% sure I built from Windows properly (getting a gomote to download and build was fiddly), but in my binaries, the first difference is in internal/buildcfg.goarm.

When built from Windows:

TEXT internal/buildcfg.goarm(SB) internal/buildcfg/cfg.go
  cfg.go:71             0x86ea00                493b6610                CMPQ 0x10(R14), SP                      
  cfg.go:71             0x86ea04                0f86c7000000            JBE 0x86ead1                            
  cfg.go:71             0x86ea0a                4883ec30                SUBQ $0x30, SP                          
  cfg.go:71             0x86ea0e                48896c2428              MOVQ BP, 0x28(SP)                       
  cfg.go:71             0x86ea13                488d6c2428              LEAQ 0x28(SP), BP                       
  cfg.go:77             0x86ea18                488d05a56a0801          LEAQ go:string.*+8348(SB), AX           
  cfg.go:50             0x86ea1f                bb05000000              MOVL $0x5, BX                           
  cfg.go:50             0x86ea24                e85707c9ff              CALL os.Getenv(SB)                      
  cfg.go:50             0x86ea29                4885db                  TESTQ BX, BX                            
  cfg.go:50             0x86ea2c                7512                    JNE 0x86ea40                            
  cfg.go:50             0x86ea2e                488d0573f34a01          LEAQ runtime.gcbits.*+2488(SB), AX      
  cfg.go:50             0x86ea35                bb01000000              MOVL $0x1, BX                           
  cfg.go:50             0x86ea3a                660f1f440000            NOPW 0(AX)(AX*1)                        
  cfg.go:78             0x86ea40                4883fb01                CMPQ $0x1, BX                           
  cfg.go:78             0x86ea44                753c                    JNE 0x86ea82                            
  cfg.go:78             0x86ea46                803835                  CMPB $0x35, 0(AX)                       
  cfg.go:78             0x86ea49                7428                    JE 0x86ea73                             
  cfg.go:80             0x86ea4b                803836                  CMPB $0x36, 0(AX)                       
  cfg.go:80             0x86ea4e                7414                    JE 0x86ea64                             
  cfg.go:82             0x86ea50                803837                  CMPB $0x37, 0(AX)                       
  cfg.go:82             0x86ea53                752d                    JNE 0x86ea82                            
  cfg.go:83             0x86ea55                b807000000              MOVL $0x7, AX                           
  cfg.go:83             0x86ea5a                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:83             0x86ea5f                4883c430                ADDQ $0x30, SP                          
  cfg.go:83             0x86ea63                c3                      RET                                     
  cfg.go:81             0x86ea64                b806000000              MOVL $0x6, AX                           
  cfg.go:81             0x86ea69                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:81             0x86ea6e                4883c430                ADDQ $0x30, SP                          
  cfg.go:81             0x86ea72                c3                      RET                                     
  cfg.go:79             0x86ea73                b805000000              MOVL $0x5, AX                           
  cfg.go:79             0x86ea78                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:79             0x86ea7d                4883c430                ADDQ $0x30, SP                          
  cfg.go:79             0x86ea81                c3                      RET                                     
  cfg.go:85             0x86ea82                488d050d960b01          LEAQ go:string.*+216174(SB), AX         
  cfg.go:85             0x86ea89                bb1e000000              MOVL $0x1e, BX                          
  cfg.go:85             0x86ea8e                31c9                    XORL CX, CX                             
  cfg.go:85             0x86ea90                31ff                    XORL DI, DI                             
  cfg.go:85             0x86ea92                4889fe                  MOVQ DI, SI                             
  cfg.go:85             0x86ea95                e80688c9ff              CALL fmt.Errorf(SB)                     
  cfg.go:85             0x86ea9a                4889058f1f1902          MOVQ AX, internal/buildcfg.Error(SB)    
  cfg.go:85             0x86eaa1                833d484d1f0200          CMPL $0x0, runtime.writeBarrier(SB)     
  cfg.go:85             0x86eaa8                7509                    JNE 0x86eab3                            
  cfg.go:85             0x86eaaa                48891d871f1902          MOVQ BX, internal/buildcfg.Error+8(SB)  
  cfg.go:85             0x86eab1                eb0c                    JMP 0x86eabf                            
  cfg.go:85             0x86eab3                488d3d7e1f1902          LEAQ internal/buildcfg.Error+8(SB), DI  
  cfg.go:85             0x86eaba                e8a129c0ff              CALL runtime.gcWriteBarrierBX(SB)       
  cfg.go:86             0x86eabf                b907000000              MOVL $0x7, CX                           
  cfg.go:86             0x86eac4                0fb6c1                  MOVZX CL, AX                            
  cfg.go:86             0x86eac7                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:86             0x86eacc                4883c430                ADDQ $0x30, SP                          
  cfg.go:86             0x86ead0                c3                      RET                                     
  cfg.go:71             0x86ead1                e88a06c0ff              CALL runtime.morestack_noctxt.abi0(SB)  
  cfg.go:71             0x86ead6                e925ffffff              JMP internal/buildcfg.goarm(SB)         
  :-1                   0x86eadb                cc                      INT $0x3                                
  :-1                   0x86eadc                cc                      INT $0x3                                
  :-1                   0x86eadd                cc                      INT $0x3                                
  :-1                   0x86eade                cc                      INT $0x3                                
  :-1                   0x86eadf                cc                      INT $0x3 

When built from Linux:

TEXT internal/buildcfg.goarm(SB) internal/buildcfg/cfg.go
  cfg.go:71             0x86ea00                493b6610                CMPQ 0x10(R14), SP                              
  cfg.go:71             0x86ea04                0f8621010000            JBE 0x86eb2b                                    
  cfg.go:71             0x86ea0a                4883ec38                SUBQ $0x38, SP                                  
  cfg.go:71             0x86ea0e                48896c2430              MOVQ BP, 0x30(SP)                               
  cfg.go:71             0x86ea13                488d6c2430              LEAQ 0x30(SP), BP                               
  cfg.go:73             0x86ea18                488b0d61201902          MOVQ internal/buildcfg.GOOS(SB), CX             
  cfg.go:73             0x86ea1f                48833d6120190207        CMPQ $0x7, internal/buildcfg.GOOS+8(SB)         
  cfg.go:73             0x86ea27                753d                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea29                8139616e6472            CMPL $0x72646e61, 0(CX)                         
  cfg.go:73             0x86ea2f                7535                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea31                668179046f69            CMPW $0x696f, 0x4(CX)                           
  cfg.go:73             0x86ea37                752d                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea39                80790664                CMPB $0x64, 0x6(CX)                             
  cfg.go:73             0x86ea3d                7527                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea3f                488b0d0a201902          MOVQ internal/buildcfg.GOARCH(SB), CX           
  cfg.go:73             0x86ea46                48833d0a20190203        CMPQ $0x3, internal/buildcfg.GOARCH+8(SB)       
  cfg.go:73             0x86ea4e                7516                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea50                6681396172              CMPW $0x7261, 0(CX)                             
  cfg.go:73             0x86ea55                750f                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea57                8079026d                CMPB $0x6d, 0x2(CX)                             
  cfg.go:73             0x86ea5b                7509                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea5d                488d0544f34a01          LEAQ runtime.gcbits.*+2488(SB), AX              
  cfg.go:73             0x86ea64                eb07                    JMP 0x86ea6d                                    
  cfg.go:73             0x86ea66                488d05e3f24a01          LEAQ runtime.gcbits.*+2400(SB), AX              
  cfg.go:77             0x86ea6d                4889442428              MOVQ AX, 0x28(SP)                               
  cfg.go:50             0x86ea72                bb05000000              MOVL $0x5, BX                                   
  cfg.go:50             0x86ea77                488d05466a0801          LEAQ go:string.*+8348(SB), AX                   
  cfg.go:50             0x86ea7e                6690                    NOPW                                            
  cfg.go:50             0x86ea80                e8fb06c9ff              CALL os.Getenv(SB)                              
  cfg.go:50             0x86ea85                4885db                  TESTQ BX, BX                                    
  cfg.go:50             0x86ea88                750a                    JNE 0x86ea94                                    
  cfg.go:77             0x86ea8a                488b442428              MOVQ 0x28(SP), AX                               
  cfg.go:77             0x86ea8f                bb01000000              MOVL $0x1, BX                                   
  cfg.go:78             0x86ea94                4883fb01                CMPQ $0x1, BX                                   
  cfg.go:78             0x86ea98                753c                    JNE 0x86ead6                                    
  cfg.go:78             0x86ea9a                803835                  CMPB $0x35, 0(AX)                               
  cfg.go:78             0x86ea9d                7428                    JE 0x86eac7                                     
  cfg.go:80             0x86ea9f                803836                  CMPB $0x36, 0(AX)                               
  cfg.go:80             0x86eaa2                7414                    JE 0x86eab8                                     
  cfg.go:82             0x86eaa4                803837                  CMPB $0x37, 0(AX)                               
  cfg.go:82             0x86eaa7                752d                    JNE 0x86ead6                                    
  cfg.go:83             0x86eaa9                b807000000              MOVL $0x7, AX                                   
  cfg.go:83             0x86eaae                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:83             0x86eab3                4883c438                ADDQ $0x38, SP                                  
  cfg.go:83             0x86eab7                c3                      RET                                             
  cfg.go:81             0x86eab8                b806000000              MOVL $0x6, AX                                   
  cfg.go:81             0x86eabd                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:81             0x86eac2                4883c438                ADDQ $0x38, SP                                  
  cfg.go:81             0x86eac6                c3                      RET                                             
  cfg.go:79             0x86eac7                b805000000              MOVL $0x5, AX                                   
  cfg.go:79             0x86eacc                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:79             0x86ead1                4883c438                ADDQ $0x38, SP                                  
  cfg.go:79             0x86ead5                c3                      RET                                             
  cfg.go:85             0x86ead6                488d05b9950b01          LEAQ go:string.*+216174(SB), AX                 
  cfg.go:85             0x86eadd                bb1e000000              MOVL $0x1e, BX                                  
  cfg.go:85             0x86eae2                31c9                    XORL CX, CX                                     
  cfg.go:85             0x86eae4                31ff                    XORL DI, DI                                     
  cfg.go:85             0x86eae6                4889fe                  MOVQ DI, SI                                     
  cfg.go:85             0x86eae9                e8b287c9ff              CALL fmt.Errorf(SB)                             
  cfg.go:85             0x86eaee                4889053b1f1902          MOVQ AX, internal/buildcfg.Error(SB)            
  cfg.go:85             0x86eaf5                833df44c1f0200          CMPL $0x0, runtime.writeBarrier(SB)             
  cfg.go:85             0x86eafc                7509                    JNE 0x86eb07                                    
  cfg.go:85             0x86eafe                48891d331f1902          MOVQ BX, internal/buildcfg.Error+8(SB)          
  cfg.go:85             0x86eb05                eb0c                    JMP 0x86eb13                                    
  cfg.go:85             0x86eb07                488d3d2a1f1902          LEAQ internal/buildcfg.Error+8(SB), DI          
  cfg.go:85             0x86eb0e                e84d29c0ff              CALL runtime.gcWriteBarrierBX(SB)               
  cfg.go:86             0x86eb13                488b4c2428              MOVQ 0x28(SP), CX                               
  cfg.go:86             0x86eb18                0fb609                  MOVZX 0(CX), CX     
  cfg.go:86             0x86eb1b                83c1d0                  ADDL $-0x30, CX                                 
  cfg.go:86             0x86eb1e                0fb6c1                  MOVZX CL, AX                                    
  cfg.go:86             0x86eb21                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:86             0x86eb26                4883c438                ADDQ $0x38, SP                                  
  cfg.go:86             0x86eb2a                c3                      RET                                             
  cfg.go:71             0x86eb2b                e83006c0ff              CALL runtime.morestack_noctxt.abi0(SB)          
  cfg.go:71             0x86eb30                e9cbfeffff              JMP internal/buildcfg.goarm(SB)                 
  :-1                   0x86eb35                cc                      INT $0x3                                        
  :-1                   0x86eb36                cc                      INT $0x3                                        
  :-1                   0x86eb37                cc                      INT $0x3                                        
  :-1                   0x86eb38                cc                      INT $0x3                                        
  :-1                   0x86eb39                cc                      INT $0x3                                        
  :-1                   0x86eb3a                cc                      INT $0x3                                        
  :-1                   0x86eb3b                cc                      INT $0x3                                        
  :-1                   0x86eb3c                cc                      INT $0x3                                        
  :-1                   0x86eb3d                cc                      INT $0x3                                        
  :-1                   0x86eb3e                cc                      INT $0x3                                        
  :-1                   0x86eb3f                cc                      INT $0x3  

It looks like on Windows build the GOOS and GOARCH checks were deadcode-eliminated, but not on the Linux build?

@prattmic
Copy link
Member

(That deadcode elimination doesn't make sense to me. GOOS and GOARCH are clearly not compile-time constants, as they are dependent on an environment variable)

@thanm

This comment was marked as outdated.

1 similar comment
@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

I poked at this a little as well, but I think the difference with the gomote appears to be due to differences in zbootstrap.go:

$ diff  from-gomote/zbootstrap.go /x/go1.20/src/internal/buildcfg/zbootstrap.go 
9c9
< const defaultGOARM = `7`
---
> const defaultGOARM = `5`

@bcmills

This comment was marked as outdated.

@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

@thanm, none of these examples are on arm, and at any rate the value of the default GOARM value shouldn't affect the cmd/test2json binary (nor should they affect the compiled github.com/ethereum/go-ethereum/cmd/geth binary AFAICT).

@jrick
Copy link
Contributor Author

jrick commented Apr 10, 2023

I can reproduce the lack of cross-platform reproducibility for commands in GOROOT too:

I believe this is because -buildid= isn't used here? We also saw issues with reproducibility when attempting to remove this (my understanding is that it should still be reproducible with it unset). Except for geth (and other programs importing the eth libraries), the builds are reproducible with the buildid set to the zero string.

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

@jrick, users should not need to set any -ldflags explicitly in order to get a reproducible build. But I did forget to set -trimpath when comparing cmd/test2json. 😅

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

...but the difference does hold with -trimpath set:

bryan@guthrie:~/go$ ./bin/go install -trimpath cmd/test2json

bryan@guthrie:~/go$ ./bin/go version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:~/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
947b411aa1eb61f8ad347df0be53e32b  ./pkg/tool/windows_amd64/test2json.exe
bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe install -trimpath cmd/test2json

bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:/mnt/c/Users/bryan/src/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
ef4b5563e65bf0bdfea675967156f55d  ./pkg/tool/windows_amd64/test2json.exe

@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

@thanm, none of these examples are on arm, and at any rate the value of the default GOARM value shouldn't affect the cmd/test2json binary (nor should they affect the compiled github.com/ethereum/go-ethereum/cmd/geth binary AFAICT).

Executable does contain a copy of the "goarm" function, even if it is not used/relevant. Linking without "-s -w":

 $ objdump -t geth.exe | fgrep goarm
[11898](sec  1)(fl 0x00)(ty  308)(scl   2) (nx 0) 0x000000000046da80 internal/buildcfg.goarm
 $

When zbootstrap.go contains "defaultGoArm = 7" (which is what you get when running make.bat on windows) then the deadcoding that Michael described is legal.

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

I still don't think that explains it? I don't see a difference in zbootstrap.go between my Windows and Linux GOROOTs, and that wouldn't affect cmd/test2json anyway.

bryan@guthrie:~/go$ diff -r ./src /mnt/c/Users/bryan/src/go/src
diff '--color=auto' -r ./src/cmd/cgo/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/cgo/zdefaultcc.go
18c18,22
<       return "c++"
---
>       switch goos {
>       case "darwin", "ios", "freebsd", "openbsd":
>               return "clang++"
>       }
>       return "g++"
diff '--color=auto' -r ./src/cmd/go/internal/cfg/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/go/internal/cfg/zdefaultcc.go
18c18,22
<       return "c++"
---
>       switch goos {
>       case "darwin", "ios", "freebsd", "openbsd":
>               return "clang++"
>       }
>       return "g++"
Only in /mnt/c/Users/bryan/src/go/src/cmd: .vscode
diff '--color=auto' -r ./src/go/build/zcgo.go /mnt/c/Users/bryan/src/go/src/go/build/zcgo.go
5c5
< const defaultCGO_ENABLED = ""
---
> const defaultCGO_ENABLED = "0"

@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

I agree that with test2json , differences in goarm can't be the culprit.

@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

I don't see a difference in zbootstrap.go between my Windows and Linux GOROOTs

Strange. Perhaps this is some sort of artifact of how gomotes work? What is your "defaultGoArm" setting, just out of curiosity?

@bcmills
Copy link
Member

bcmills commented Apr 10, 2023

Strange. Perhaps this is some sort of artifact of how gomotes work? What is your "defaultGoArm" setting, just out of curiosity?

My zbootstrap.go has

const defaultGOARM = `7`

But IIUC the defaultGOARM is only supposed to vary when compiling ARM on ARM, so it should also be 7 on your machine unless you're somehow using an arm32 as your development host. 😅

@prattmic
Copy link
Member

prattmic commented Apr 10, 2023

Under what circumstances do we rewrite zbootstrap.go? I also have defaultGOARM = "5" in zbootstrap.go in my main Go checkout, which has been around for years. In another, newer, checkout, I have defaultGOARM = "7".

Edit: Ignore above, that is because this behavior has changed since 1.20 in https://go.dev/cl/470695.

@thanm
Copy link
Contributor

thanm commented Apr 10, 2023

Thanks for that (CL 470695). Yeah, building on linux from 1.20 release branch I get "defaultGOARM" of 5.

@bcmills
Copy link
Member

bcmills commented Apr 11, 2023

@jrick, is that using objdump or go tool objdump? The latter supports Windows executables, but the former (on Linux) generally does not.

@jrick
Copy link
Contributor Author

jrick commented Apr 11, 2023

it was go tool objdump

I see this diffing the go tool nm =sort=address output:

  147e560 T github.com/ethereum/go-ethereum/common/lru.(*Cache[go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.number uint64; github.com/ethereum/go-ethereum/eth/gasprice.percentiles string },go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.reward []*math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.baseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.nextBaseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.gasUsedRatio float64 }]).Contains.func1
  147e5c0 T github.com/ethereum/go-ethereum/common/lru.(*Cache[go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.number uint64; github.com/ethereum/go-ethereum/eth/gasprice.percentiles string },go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.reward []*math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.baseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.nextBaseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.gasUsedRatio float64 }]).Add.func1
  147e620 T main.init.0.func1
  147e660 T main.init.0.func2
  147e6a0 T main.init
  1484060 T main.importChain.func2
  14840c0 T main.prepare.func1
  1484120 T type:.eq.main.snapshotIterator
  14841c0 T github.com/ethereum/go-ethereum/ethdb.Database.Get-fm
  1484233 T runtime.etext
  1485000 R runtime.rodata
  1485000 R runtime.types
  1485000 R type:*
  18590a8 R go:string.*
- 1abc148 R go:func.*
- 1c7e138 R runtime.gcbits.*
- 1c81e38 R runtime.gcbss
- 1c82b0a R runtime.gcdata
- 1c82b0a R runtime.egcbss
- 1c83948 R $f32.35800000...

@jrick
Copy link
Contributor Author

jrick commented Apr 11, 2023

The closest difference i can find is in runtime.types (but it didn't affect the address of anything else before the minus parts of the diff).

--- a   2023-04-11 17:36:30.634377706 -0400
+++ b   2023-04-11 17:36:34.694376931 -0400
@@ -61,7 +61,7 @@
   type.go:66           0x462a13                4883c6ef                ADDQ $-0x11, SI
   type.go:66           0x462a17                4883fe08                CMPQ $0x8, SI
   type.go:66           0x462a1b                773a                    JA 0x462a57
-  type.go:66           0x462a1d                488d0dfccc8301          LEAQ runtime.typesEqual.jump15(SB), CX
+  type.go:66           0x462a1d                488d0d3c058401          LEAQ runtime.typesEqual.jump15(SB), CX
   type.go:66           0x462a24                ff24f1                  JMP 0(CX)(SI*8)
   type.go:96           0x462a27                488d7048                LEAQ 0x48(AX), SI
   type.go:593          0x462a2b                eb2e                    JMP 0x462a5b
@@ -91,7 +91,7 @@
   type.go:66           0x462a7c                4983c0ef                ADDQ $-0x11, R8
   type.go:66           0x462a80                4983f808                CMPQ $0x8, R8
   type.go:66           0x462a84                773c                    JA 0x462ac2
-  type.go:66           0x462a86                488d0df3cc8301          LEAQ runtime.typesEqual.jump30(SB), CX
+  type.go:66           0x462a86                488d0d33058401          LEAQ runtime.typesEqual.jump30(SB), CX
   type.go:66           0x462a8d                42ff24c1                JMP 0(CX)(R8*8)
   type.go:96           0x462a91                4c8d4748                LEAQ 0x48(DI), R8
   type.go:594          0x462a95                eb2f                    JMP 0x462ac6
@@ -160,7 +160,7 @@
   type.go:608          0x462b86                4883c6ef                ADDQ $-0x11, SI
   type.go:608          0x462b8a                4883fe09                CMPQ $0x9, SI
   type.go:608          0x462b8e                0f879c080000            JA 0x463430
-  type.go:608          0x462b94                488d0d85e48301          LEAQ runtime.typesEqual.jump59(SB), CX
+  type.go:608          0x462b94                488d0dc51c8401          LEAQ runtime.typesEqual.jump59(SB), CX
   type.go:608          0x462b9b                ff24f1                  JMP 0(CX)(SI*8)
   type.go:614          0x462b9e                488b5030                MOVQ 0x30(AX), DX
   type.go:614          0x462ba2                488b5f30                MOVQ 0x30(DI), BX

@bcmills bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 11, 2023
@rsc
Copy link
Contributor

rsc commented Apr 11, 2023

The goarm-related inconsistencies are definitely present in Go 1.20 and fixed for Go 1.21 already.

@rsc
Copy link
Contributor

rsc commented Apr 11, 2023

Worth pointing to my comment #59525 (comment), which applies here too. cmd/go imports go/build, and it reproduces with the setup described there.

The original report is because of a bug in Go 1.20, which has been fixed.

Reports of problems in the current devel tree are likely to turn out to be caused by other incidentals, such as doing this from a git checkout (as opposed to a clean source tree, because of embedded VCS information) or accidentally leaving a few files behind in one computer versus the other. By far the best way to set up the test environment is to make a tgz of your source tree (or use ./make.bash -distpack to get one) and then copy it to the multiple test machines, unpack in a clean directory, and ./make.bash from there.

It is true that CGO_ENABLED=0 ./make.bash and ./make.bash (without CGO_ENABLED set) produce toolchains that produce different binaries in some builds. The same is true for GOARM=5 ./make.bash vs ./make.bash. That's essentially unavoidable, since those details appear in the way go/build.Default is populated, and we don't want to break that behavior. The answer is to only use plain ./make.bash if you want the canonical toolchain to use for reproducible builds. (Again, there were some problems in this regard in Go 1.20, but they are fixed and will ship in Go 1.21.)

@jrick
Copy link
Contributor Author

jrick commented Apr 11, 2023

So coming from #59525, I am now running a development version of Go with VERSION file. test2json is reproducible now (without clearing the build ID), but geth remains unreproducible.

@rsc
Copy link
Contributor

rsc commented Apr 11, 2023

Does ./make.bash -distpack / ./make.bat -distpack reproduce (same sha256 hashes printed) on the two machines?

@rsc
Copy link
Contributor

rsc commented Apr 11, 2023

If the two do not reproduce, then the next thing to check would be any Go-relevant environment variables (like CGO_ENABLED or GOARM) being set or unset differently on the two machines. GOOS, GOARCH, and GOROOT are okay to be different, of course.

@jrick
Copy link
Contributor Author

jrick commented Apr 11, 2023

./make.bash -distpack with CL 483695 cherrypicked over master fails to build the distpack, so I think something needs to be fixed there before it is merged to master.

./make.bash -distpack
Building Go cmd/dist using /usr/lib64/go/1.20. (go1.20.3 linux/amd64)
Building Go toolchain1 using /usr/lib64/go/1.20.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
Packaging archives for linux/amd64.
distpack: unexpected source archive file: go/src/internal/platform/zcgo.go (generated by dist)
go tool dist: FAILED: /home/jrick/src/go/pkg/tool/linux_amd64/distpack: exit status 1

Building master (at 1635205) without the CL shows different hashes for all of the dist files. CGO_ENABLED is not set for either, and the only GO* environment variables that are set differently is GOPATH=C:\Users\jrick\go in my windows environment. (I did not set this,; I think the msi installer did.) Unsetting GOPATH does not change the distpack hashes.

@jrick
Copy link
Contributor Author

jrick commented Apr 12, 2023

Well I figured out why that was. I had a stray test2json.exe in the root of my Go source tree, which made its way into the src dist. Now the src dist matches exactly.

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

What about geth.exe? I was able to build it reproducibly for windows/amd64 on windows/amd64 and linux/amd64 machines. I used

GOOS=windows go build -trimpath -ldflags=-w -o geth.exe ./cmd/geth

@jrick
Copy link
Contributor Author

jrick commented Apr 12, 2023

Nope, still getting different geth.exe builds with that exact command. The diff looks the same as #59500 (comment).

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

Now that the src archives match, the next thing to check is

./make.bat -distpack  (Windows)
GOOS=windows ./make.bash -distpack  (Linux)

Those need to print all the same hashes. Once those match, we can move on to Ethereum differences. I am building from a copy of go-ethereum I built using git archive and then extracted into a clean directory, so there's no .git during the build. Using the same tgz on both systems also helps me make sure the source code tree really is the same in both places. I was using specifically https://swtch.com/tmp/eth59500.tgz.

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

(If you can get a reproducible build from the tgz but not from a checkout then maybe you need -buildvcs=false or maybe something else needs more examination. But trying the same tgz on both machines is probably the right next step.)

@jrick
Copy link
Contributor Author

jrick commented Apr 12, 2023

All of the distpack hashes are matching when I build for windows from linux.

I used this VERSION:

go1.999999
time 2023-04-12T00:00:00Z

with Go master checked out at 4b154e5.

Packaging archives for linux/amd64.
distpack: 7aa3c6f5cfb61463 go1.999999.src.tar.gz
distpack: 7fce11ee2a7dda85 go1.999999.linux-amd64.tar.gz
distpack: f0f67810994a601d v0.0.1-go1.999999.linux-amd64.zip
distpack: 58528cce1848ddf4 v0.0.1-go1.999999.linux-amd64.mod
distpack: 298e9120095addc3 v0.0.1-go1.999999.linux-amd64.info

Packaging archives for windows/amd64.
distpack: 7aa3c6f5cfb61463 go1.999999.src.tar.gz
distpack: 97be81cee48e264b go1.999999.windows-amd64.zip
distpack: bfe37943e94852ae v0.0.1-go1.999999.windows-amd64.zip
distpack: 58528cce1848ddf4 v0.0.1-go1.999999.windows-amd64.mod
distpack: d14c788d7d9c1355 v0.0.1-go1.999999.windows-amd64.info

But using that .tgz, I'm still getting different geth.exe hashes:

$ ~/src/go/bin/go version && GOOS=windows ~/src/go/bin/go build -trimpath -ldflags=-w -o geth.exe ./cmd/geth && openssl sha256 ./geth.exe
go version go1.999999 linux/amd64
SHA2-256(./geth.exe)= 39e229d1fb243247a7057ebcca1341971398d4cb7429d9b0f6be16f18ac4cd24

$ ~/src/go/bin/go version && GOOS=windows ~/src/go/bin/go build -trimpath -ldflags=-w -o geth.exe ./cmd/geth && openssl sha256 ./geth.exe
go version go1.999999 windows/amd64
SHA256(./geth.exe)= 8ed3b57cfd55a7024e7789cc107c7b9e5751614cfcd8bb520d2622fb4447e23a

@davecgh
Copy link

davecgh commented Apr 12, 2023

I'm seeing basically the same results as @jrick. The linux hash of geth matches, but the one built on Windows neither matches the one on Linux nor the one in @jrick's comment.

From linux:

$ git rev-parse HEAD
4b154e55baced13a792ebd6a3796b31f49938a71

$ cat VERSION
go1.999999
time 2023-04-12T00:00:00Z

$ GOOS=windows ./make.bash -distpack
...
Packaging archives for windows/amd64.
distpack: 7aa3c6f5cfb61463 go1.999999.src.tar.gz
distpack: 97be81cee48e264b go1.999999.windows-amd64.zip
distpack: bfe37943e94852ae v0.0.1-go1.999999.windows-amd64.zip
distpack: 58528cce1848ddf4 v0.0.1-go1.999999.windows-amd64.mod
distpack: d14c788d7d9c1355 v0.0.1-go1.999999.windows-amd64.info

From windows (under msys2):

$ git rev-parse HEAD
4b154e55baced13a792ebd6a3796b31f49938a71

$ cat VERSION
go1.999999
time 2023-04-12T00:00:00Z

$ ./make.bat -distpack
...
Packaging archives for windows/amd64.
distpack: 7aa3c6f5cfb61463 go1.999999.src.tar.gz
distpack: 97be81cee48e264b go1.999999.windows-amd64.zip
distpack: bfe37943e94852ae v0.0.1-go1.999999.windows-amd64.zip
distpack: 58528cce1848ddf4 v0.0.1-go1.999999.windows-amd64.mod
distpack: d14c788d7d9c1355 v0.0.1-go1.999999.windows-amd64.info

Then building geth:

$ go version && GOOS=windows go build -trimpath -ldflags=-w -o geth.exe ./cmd/geth && openssl sha256 ./geth.exe
go version go1.999999 linux/amd64
SHA2-256(./geth.exe)= 39e229d1fb243247a7057ebcca1341971398d4cb7429d9b0f6be16f18ac4cd24

$ go version && GOOS=windows go build -trimpath -ldflags=-w -o geth.exe ./cmd/geth && openssl sha256 ./geth.exe
go version go1.999999 windows/amd64
SHA2-256(./geth.exe)= 41417d47bacfc4afad8cdee64a1533bc846449ed4cdfd35efc7a3c6c414fa3f4

@bcmills
Copy link
Member

bcmills commented Apr 12, 2023

@jrick, @davecgh: are you building from the same clone of the repo in both cases? (Are the repo contents identical?)

Also, if you set -buildvcs=false, does reproducibility improve? (Perhaps Git for Windows produces different metadata from the mainline Git.)

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

I have reproduced a non-reproducibility problem building geth. Will investigate.

@jrick
Copy link
Contributor Author

jrick commented Apr 12, 2023

I used separate source trees, but if I build inside WSL from the same source tree that I used for the native windows build (building in /mnt/c/...), I get the same hash as from the source tree on the Linux FS.

@rsc rsc changed the title cmd/dist,go/build: files generated by dist cause non-reproducible builds for binaries that import go/build cmd/go: ethereum build not reproducible Apr 12, 2023
@randall77
Copy link
Contributor

Possibly related to #59571

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

Yes, urfave/cli is the root cause for geth too.

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

Once #59571 is fixed, all evidence points to this being fixed too. We can leave it open until then and see.

@jrick
Copy link
Contributor Author

jrick commented Apr 13, 2023

I am able to build geth reproducibly using the same source checkout (the extracted tgz), with the Linux test using the same files as Windows through the WSL /mnt/c/ path.

However, I have not been able to create a reproducible geth.exe when building from two different go-ethereum git checkouts at same commit, with clean source trees. I double checked my Git for Windows settings, and autocrlf is false. If I do the Linux build from the same Git source tree as windows, that ends up being reproducible.

So I checked the sha256 hashes of the Go source files in the go-ethereum checkouts and they are all different. 😕

Not sure what is going on with that, but seems like the Go reproducibility issues are solved.

@jrick
Copy link
Contributor Author

jrick commented Apr 13, 2023

Seems to be the line endings

windows:

00000000: 2f2f 2043 6f70 7972 6967 6874 2032 3032  // Copyright 202
00000010: 3220 5468 6520 676f 2d65 7468 6572 6575  2 The go-ethereu
00000020: 6d20 4175 7468 6f72 730d 0a2f 2f20 5468  m Authors..// Th

linux:

00000000: 2f2f 2043 6f70 7972 6967 6874 2032 3032  // Copyright 202
00000010: 3220 5468 6520 676f 2d65 7468 6572 6575  2 The go-ethereu
00000020: 6d20 4175 7468 6f72 730a 2f2f 2054 6869  m Authors.// Thi

@rsc
Copy link
Contributor

rsc commented Apr 13, 2023

Amazing how CRLFs are still a problem in 2023, but I'm glad you solved everything else. Thanks for confirming.

@rsc rsc closed this as completed Apr 13, 2023
gopherbot pushed a commit that referenced this issue Apr 20, 2023
During bootstrapping, cmd/dist writes a file indicating which
GOOS/GOARCH combinations are valid, and which support cgo-enabled
builds. That information previously went into the go/build package,
but today it fits in more naturally in the internal/platform package
(which already has a number of functions indicating feature support
for GOOS/GOARCH combinations).

Moreover, as of CL 450739 the cmd/go logic for determining whether to
use cgo is somewhat more nuanced than the go/build logic: cmd/go
checks for the presence of a C compiler, whereas go/build does not
(mostly because it determines its configuration at package-init time,
and checking $PATH for a C compiler is somewhat expensive).

To simplify this situation, this change:

	- consolidates the “cgo supported” check in internal/platform
	  (alongside many other platform-support checks) instead of making
	  it a one-off in go/build,

	- and updates a couple of tests to use testenv.HasCGO instead of
	  build.Default.CgoEnabled to decide whether to test a cgo-specific
	  behavior.

For #58884.
For #59500.

Change-Id: I0bb2502dba4545a3d98c9e915727382ce536a0f3
Reviewed-on: https://go-review.googlesource.com/c/go/+/483695
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

10 participants