Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go build -i cache mixup with GOARM=6 vs GOARM=7 #41223

Open
ncw opened this issue Sep 4, 2020 · 0 comments
Open

cmd/go: go build -i cache mixup with GOARM=6 vs GOARM=7 #41223

ncw opened this issue Sep 4, 2020 · 0 comments
Milestone

Comments

@ncw
Copy link
Contributor

@ncw ncw commented Sep 4, 2020

What version of Go are you using (go version)?

go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ncw/.cache/go-build"
GOENV="/home/ncw/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ncw/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ncw/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/go/go1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/go/go1.15/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build829175078=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I checked out the rclone repository: https://github.com/rclone/rclone - probably any large repo which builds under ARM will do.

I then ran this script

#!/bin/bash

echo "sequential build"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -i -o sequential-i-v6.bin -trimpath 2>sequential-i-v6.log
GOOS=linux GOARCH=arm GOARM=7 go build -v -i -o sequential-i-v7.bin -trimpath 2>sequential-i-v7.log

echo "parallel build with -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -i -o parallel-i-v6.bin -trimpath 2>parallel-i-v6.log &
GOOS=linux GOARCH=arm GOARM=7 go build -v -i -o parallel-i-v7.bin -trimpath 2>parallel-i-v7.log &

wait

echo "sequential build without -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -o sequential-no-i-v6.bin -trimpath 2>sequential-no-i-v6.log
GOOS=linux GOARCH=arm GOARM=7 go build -v -o sequential-no-i-v7.bin -trimpath 2>sequential-no-i-v6.log

echo "parallel build without -i"
rm -rf ~/go/pkg/linux_arm
go clean -cache
GOOS=linux GOARCH=arm GOARM=6 go build -v -o parallel-no-i-v6.bin -trimpath 2>parallel-no-i-v6 &
GOOS=linux GOARCH=arm GOARM=7 go build -v -o parallel-no-i-v7.bin -trimpath 2>parallel-no-i-v7 &

wait

md5sum *.bin | sort

echo "done"

What did you expect to see?

I expected to see 8 binaries with 2 md5sums

What did you see instead?

I see that the binaries built in parallel with go build -i appear to be different

Note that the differing binaries sometimes differ from run to run.

run 1

1160905621e7ce72dd577d7cbbffc3b2  parallel-no-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-no-i-v7.bin
f2c7c50467ab0ec6354ec8f35805b65b  parallel-i-v7.bin

c048804419a985ba1f5bde7c7f2dd47f  parallel-no-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-no-i-v6.bin
126979dfe3987ad89a18af57f4089b02  parallel-i-v6.bin

run 2-3

Identical to above

run 4

1160905621e7ce72dd577d7cbbffc3b2  parallel-no-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-i-v7.bin
1160905621e7ce72dd577d7cbbffc3b2  sequential-no-i-v7.bin
6a99ec2ac3e4ea4684882da699ac9d7b  parallel-i-v7.bin

c048804419a985ba1f5bde7c7f2dd47f  parallel-no-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-i-v6.bin
c048804419a985ba1f5bde7c7f2dd47f  sequential-no-i-v6.bin
92e501e7ef370324003c0100ce49ea6f  parallel-i-v6.bin

Discussion

This came to my attention in rclone/rclone#4553 - ARMv6 builds for 1.53.0 result in "Illegal instruction" error.

The root cause of that issue seems to be rclone's builder building the ARMv6 and ARMv7 releases in parallel and somehow parts of the ARMv7 binary get mixed into the ARMv6 binary resulting in the illegal instruction on ARMv6. However the binary does run on ARMv7 indicating that it isn't just random corruption, it is probably compilation units that have got mixed up.

Rclone's builder is using go build -i for historical reasons (namely that is how you used to get build caching). Removing the -i from the command appears to fix the problem.

ncw added a commit to rclone/rclone that referenced this issue Sep 4, 2020
Before this change we used `go build -i` to build the releases in parallel.

However this causes the ARMv6 and ARMv7 build to get mixed up somehow,
causing an illegal instruction when running rclone binaries on ARMv6.

See go bug: golang/go#41223

This removes the -i which should have no effect on build times on the
CI and appears to fix the problem.
ncw added a commit to rclone/rclone that referenced this issue Sep 4, 2020
Before this change we used `go build -i` to build the releases in parallel.

However this causes the ARMv6 and ARMv7 build to get mixed up somehow,
causing an illegal instruction when running rclone binaries on ARMv6.

See go bug: golang/go#41223

This removes the -i which should have no effect on build times on the
CI and appears to fix the problem.
@jayconrod jayconrod added this to the Backlog milestone Sep 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.