Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT #27285

Closed
vincentbernat opened this issue Aug 27, 2018 · 24 comments

Comments

Projects
None yet
10 participants
@vincentbernat
Copy link

commented Aug 27, 2018

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bernat/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bernat/src/gocode"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/bernat/tmp/reproduce/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build345235277=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Using the following main.go file:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	helloHandler := func(w http.ResponseWriter, req *http.Request) {
		io.WriteString(w, "Hello, world!\n")
	}

	http.HandleFunc("/hello", helloHandler)
	http.ListenAndServe(":8080", nil)
}

And the following commands:

go mod init anything
go build -i -o /dev/null

I get:

go: finding golang.org/x/text v0.3.0
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/crypto/poly1305: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/text/transform: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied

What did you expect to see?

go build shouldn't try to erase stuff shipped with Go. This only seems to happen for vendorized stuff. Non-vendorized stuff don't get compiled. Without -i, this doesn't happen. The provided command is the one issued by a flycheck plugin for Emacs.

@vincentbernat vincentbernat changed the title go build 1.11 tries to rebuild some vendorized dependencies go build 1.11 tries to rebuild shipped vendorized dependencies Aug 27, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

I think this is a duplicate of #26988.

@ianlancetaylor ianlancetaylor changed the title go build 1.11 tries to rebuild shipped vendorized dependencies cmd/go: go build 1.11 tries to rebuild shipped vendorized dependencies Aug 28, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

I thought #26998 was fixed by https://golang.org/cl/130138, although I see that the CL does not mention the issue.

This should be fixed, but, in the meantime, don't use go build -i. Now that we have the build cache, go build -i isn't useful.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Aug 30, 2018

@gopherbot please file this for backport against 1.11. This is a regression.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 30, 2018

Backport issue(s) opened: #27389 (for 1.11).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2018

Honestly I'm not sure it helps to file an issue for backport when we don't have a fix yet.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Aug 30, 2018

@ianlancetaylor the point of opening a backport issue is that it will not get closed when a "Fixes" patch gets merged. We used to lose track of things that way. Opening it immediately avoids having to track an extra "should backport but don't have a fix yet" state separately, and having to take a GitHub action when it transitions to "it has a fix now" by a Gerrit action.

releasebot pushes all CherryPickCandidate issues to the next minor release, so we don't even have to manage them, making them basically zero overhead.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2018

But the downside is that when we are looking at a minor release, we have to look at these issues again and again, each time going back to the original issue, and working out that it has not been fixed and there is nothing to backport.

Since it is gopherbot that is closing the issues, we don't have to be hostage to it. I suggest that if gopherbot sees that the issue has a milestone of an earlier minor release, that the issue not be closed when the patch is committed to tip.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Sep 5, 2018

@ianlancetaylor It's GitHub, not gopherbot, that closes issues when a patch is submitted to master. (gopherbot only takes care of merges to release branches.) The other issue is when something needs backport to two releases, which we also used to lose track of. It sounds like a gopherbot feature that only applies the CherryPickCandidate label when the main issue is closed would solve this? In any case, we are OT, so if you'd like to propose that or a different change, please open a separate issue.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Sep 5, 2018

More details about this rebuilding issue at #27482.

@FiloSottile FiloSottile added the modules label Sep 5, 2018

@bcmills bcmills self-assigned this Sep 6, 2018

jessepeterson added a commit to jessepeterson/micromdm that referenced this issue Sep 7, 2018

groob added a commit to micromdm/micromdm that referenced this issue Sep 7, 2018

@godwhoa

This comment has been minimized.

Copy link

commented Sep 7, 2018

For anyone running into this issue, try the current master. Fixed in my case.

➜  bug27285 go version
go version go1.11rc1 linux/amd64
➜  bug27285 go build -i
go build golang_org/x/crypto/poly1305: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/text/transform: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied

➜  bug27285 go version
go version devel +ceb7745 Fri Sep 7 18:58:57 2018 +0000 linux/amd64
➜  bug27285 go build -i
# anything
./main.go:4:2: imported and not used: "fmt"
./main.go:10:3: undefined: io
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@godwhoa, I see traces there for rc1 and devel. Have you tried the actual 1.11 release? (I believe some module bugs were fixed in between rc1 and rc2.)

@godwhoa

This comment has been minimized.

Copy link

commented Sep 7, 2018

➜  bug27285 go version
go version go1.11 linux/amd64
➜  bug27285 go build -i
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied
go build golang_org/x/crypto/poly1305: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/text/transform: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

Thanks!

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

My first clue so far is that the effective ImportPath of the vendored dependencies varies between GOPATH mode and module mode: in GOPATH mode they have the prefix vendor/, where in module mode they do not.

I suspect that that results in different artifacts and/or different cache keys.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

(And I have a test case that replicates this.)

cippaciong added a commit to cippaciong/refresh that referenced this issue Sep 29, 2018

Skip -i flag in go build if modules are enabled
Using -i flag when running go build is not useful with go modules enabled
and in some cases can lead to a bug, with go trying to rebuild
shipped vendorized dependencies as described in this issue
golang/go#27285.
With this commit we check if go modules are enabled and we drop the -i
flag accordingly.

markbates added a commit to markbates/refresh that referenced this issue Sep 29, 2018

Skip -i flag in go build if modules are enabled (#25)
Using -i flag when running go build is not useful with go modules enabled
and in some cases can lead to a bug, with go trying to rebuild
shipped vendorized dependencies as described in this issue
golang/go#27285.
With this commit we check if go modules are enabled and we drop the -i
flag accordingly.

cippaciong added a commit to cippaciong/buffalo that referenced this issue Sep 29, 2018

Skip -i flag in go build if modules are enabled
Using -i flag when running go build is not useful with go modules enabled
and in some cases can lead to a bug, with go trying to rebuild
shipped vendorized dependencies as described in this issue
golang/go#27285.
With this commit we check if go modules are enabled and we drop the -i
flag accordingly.

markbates added a commit to gobuffalo/buffalo that referenced this issue Sep 29, 2018

Skip -i flag in go build if modules are enabled (#1337)
Using -i flag when running go build is not useful with go modules enabled
and in some cases can lead to a bug, with go trying to rebuild
shipped vendorized dependencies as described in this issue
golang/go#27285.
With this commit we check if go modules are enabled and we drop the -i
flag accordingly.

phlogistonjohn added a commit to phlogistonjohn/heketi that referenced this issue Oct 4, 2018

makefile: stop using -i in go build options
After reading up a bit on go modules (vgo) it appears
that -i is no longer needed or recommended for Go [1].
It may have improved performance on some versions prior to
1.10 but is not needed for correctness and may not
have been much use for containerized builds anyway.

1: golang/go#27285 (comment)

Signed-off-by: John Mulligan <jmulligan@redhat.com>

obnoxxx added a commit to heketi/heketi that referenced this issue Oct 8, 2018

makefile: stop using -i in go build options
After reading up a bit on go modules (vgo) it appears
that -i is no longer needed or recommended for Go [1].
It may have improved performance on some versions prior to
1.10 but is not needed for correctness and may not
have been much use for containerized builds anyway.

1: golang/go#27285 (comment)

Signed-off-by: John Mulligan <jmulligan@redhat.com>
@Foxboron

This comment has been minimized.

Copy link

commented Oct 22, 2018

Is this issue possibly connected with #24034?

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 22, 2018

Possibly. The fix in both cases may be to make install a no-op for files in GOROOT.

prydie added a commit to oracle/oci-cloud-controller-manager that referenced this issue Oct 28, 2018

owainlewis added a commit to oracle/oci-cloud-controller-manager that referenced this issue Oct 29, 2018

Working Volume provisioner E2E tests (#260)
* Run VP E2E tests in CI
* Fix FSS not found bug
* Use mount target in AD2 in FSS E2E
* Use cloud-provider config in volume provisioner
* Don't use go's -i flag anymore. See: golang/go#27285
* Use cloud-provider config in flexvolume driver
@gopherbot

This comment has been minimized.

Copy link

commented Nov 5, 2018

Change https://golang.org/cl/147443 mentions this issue: vendor/golang_org: move to internal/golang_org

@bcmills bcmills changed the title cmd/go: go build 1.11 tries to rebuild shipped vendorized dependencies cmd/go: go build 1.11 tries to rebuild vendored dependencies in GOROOT Nov 15, 2018

@bcmills bcmills changed the title cmd/go: go build 1.11 tries to rebuild vendored dependencies in GOROOT cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT Nov 15, 2018

wusendong added a commit to wusendong/bk-cmdb that referenced this issue Nov 21, 2018

@gopherbot gopherbot closed this in 2012227 Nov 29, 2018

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue Apr 25, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 2, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 8, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019

hectorj2f added a commit to hectorj2f/virtual-kubelet that referenced this issue May 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.