Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: "can't find import" error due to race during concurrent build #22423

Closed
bassam opened this issue Oct 24, 2017 · 5 comments

Comments

Projects
None yet
4 participants
@bassam
Copy link

commented Oct 24, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.9.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/rook/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

on a machine with 16 cores (like a c4.4xlarge instance in EC2) I run the rook build system as follows:

git clone https://github.com/rook/rook
make distclean
build/run make -j16 build.all V=1 | tee build.log

What did you expect to see?

I expect the build to succeed every time.

What did you see instead?

Instead, I see intermittent errors that look like a race in the go build process. For example:

# github.com/rook/rook/vendor/k8s.io/api/autoscaling/v1
vendor/k8s.io/api/autoscaling/v1/types.go:20:2: can't find import: "github.com/rook/rook/vendor/k8s.io/api/core/v1"
mkdir -p /home/rook/go/pkg/linux_arm64_static/github.com/rook/rook/vendor/k8s.io/api/core/
cp $WORK/github.com/rook/rook/vendor/k8s.io/api/core/v1.a /home/rook/go/pkg/linux_arm64_static/github.com/rook/rook/vendor/k8s.io/api/core/v1.a

Note that the cp command happened after the can't find import error.

@ALTree ALTree changed the title "can't find import" error due to race during concurrent build cmd/compile: "can't find import" error due to race during concurrent build Oct 24, 2017

@bassam

This comment has been minimized.

Copy link
Author

commented Oct 24, 2017

Note that our build system runs concurrent go build from the makefile. Not sure if this is causing .a packages to be removed form the file system and resulting in errors like the can't import

@ALTree

This comment has been minimized.

Copy link
Member

commented Oct 24, 2017

Easy to test if there is an issue in your makefile: force sequential compilation in the Go compiler. Does the error go away?

@bassam

This comment has been minimized.

Copy link
Author

commented Oct 24, 2017

@ALTree with export GO19CONCURRENTCOMPILATION=0 and after 5 builds I've not seen the build error. Note however that we are running concurrent go build commands for the same platform and they might be attempting to build similar import packages. Is this a supported scenario?

Looking at the copyFile function I see that it attempts to remove the file before copying. I can't imagine this being safe to do when concurrent builds of go build are running and import the same packages.

I'm going to change our makefile to only run concurrently across build platforms, and sequentially within a build platform. I'll report back here.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2017

Running concurrent go build commands in the same GOPATH is clearly unsafe. Don't do that.

@bassam

This comment has been minimized.

Copy link
Author

commented Oct 24, 2017

@ianlancetaylor yes I see why now. Fixed our makefile to avoid that. Thanks!

@bassam bassam closed this Oct 24, 2017

@golang golang locked and limited conversation to collaborators Oct 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.