New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: `go install` with CGO_ENABLED=0 tries to install standard packages, which can fail #18981

Closed
glycerine opened this Issue Feb 7, 2017 · 15 comments

Comments

Projects
None yet
8 participants
@glycerine

glycerine commented Feb 7, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

[jaten@biggie tmp]$ go version
go version go1.8rc3 linux/amd64 ## also happens on go1.7.4
[jaten@biggie tmp]$

What operating system and processor architecture are you using (go env)?

[jaten@biggie tmp]$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/jaten/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build885638224=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="0"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
[jaten@biggie tmp]$

What did you do?

I ran make for https://github.com/glycerine/vhaline
Makefile does go install on the library github.com/glycerine/vhaline/vhaline and then on a test executable; github.com/glycerine/vhaline/cmd/vhaline

What did you expect to see?

successful build. build is successful without CGO_ENABLED=0

What did you see instead?

[jaten@biggie vhaline]$ make                                                                                                  
cd vhaline && go install && cd ../cmd/vhaline && go install                                                                   
go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied                  ### problem!                                 
make: *** [build] Error 1                                                                                                     
[jaten@biggie vhaline]$ go version                                                                                            
go version go1.8rc3 linux/amd64                                                                                               
[jaten@biggie vhaline]$ env|grep GO                                                                                           
CGO_ENABLED=0                                                                                                                 
GOROOT=/usr/local/go                                                                                                          
GODEBUG=netdns=go                                                                                                             
GOPATH=/home/jaten/go                                                                                                         
[jaten@biggie vhaline]$ ls -al /usr/local/go/pkg/linux_amd64/net.a                                                            
-rw-r--r-- 1 root root 1209910 Jan 26 18:55 /usr/local/go/pkg/linux_amd64/net.a                                               
[jaten@biggie vhaline]$ file /usr/local/go/pkg/linux_amd64/net.a                                                              
/usr/local/go/pkg/linux_amd64/net.a: current ar archive                                                                       
[jaten@biggie vhaline]$ mkdir tmp                                                                                             
[jaten@biggie vhaline]$ cd tmp                                                                                                
[jaten@biggie tmp]$ ar x /usr/local/go/pkg/linux_amd64/net.a                                                                  
[jaten@biggie tmp]$ ls -al                                                                                                    
total 1192                                                                                                                    
drwxrwxr-x 2 jaten jaten      48 Feb  7 16:57 .                                                                               
drwxrwxr-x 7 jaten jaten    4096 Feb  7 16:57 ..                                                                              
-rw-r--r-- 1 jaten jaten   19172 Feb  7 16:57 _all.o                                                                          
-rw-r--r-- 1 jaten jaten 1169185 Feb  7 16:57 _go_.o                                                                          
-rw-r--r-- 1 jaten jaten   21364 Feb  7 16:57 __.PKGDEF                                                                       
[jaten@biggie tmp]$ go env    
GOARCH="amd64"                                                                                                                
GOBIN=""                                                                                                                      
GOEXE=""                                                                                                                      
GOHOSTARCH="amd64"                                                                                                            
GOHOSTOS="linux"                                                                                                              
GOOS="linux"                                                                                                                  
GOPATH="/home/jaten/go"                                                                                                       
GORACE=""                                                                                                                     
GOROOT="/usr/local/go"                                                                                                        
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"                                                                                
GCCGO="gccgo"                                                                                                                 
CC="gcc"                                                                                                                      
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build885638224=/tmp/go-build -gno-record-gcc-switches"   
CXX="g++"                                                                                                                     
CGO_ENABLED="0"                                                                                                               
PKG_CONFIG="pkg-config"                                                                                                       
CGO_CFLAGS="-g -O2"                                                                                                           
CGO_CPPFLAGS=""                                                                                                               
CGO_CXXFLAGS="-g -O2"                                                                                                         
CGO_FFLAGS="-g -O2"                                                                                                           
CGO_LDFLAGS="-g -O2"                                                                                                          
[jaten@biggie tmp]$ go version                                                                                                
go version go1.8rc3 linux/amd64                                                                                               
[jaten@biggie tmp]$ unset CGO_ENABLED                                                                                         
[jaten@biggie tmp]$ cd ..                                                                                                     
[jaten@biggie vhaline]$ make                                                                                                  
cd vhaline && go install && cd ../cmd/vhaline && go install                                                                   
[jaten@biggie vhaline]$ #successful build  

@glycerine glycerine changed the title from linux/amd64 can't build with CGO_ENABLED=0: o install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied to linux/amd64 can't build with CGO_ENABLED=0: go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied Feb 7, 2017

@davecheney

This comment has been minimized.

Contributor

davecheney commented Feb 7, 2017

Your makefile is calling go install, which fails because your user does not have permission to write to that directory.

Either use go build to skip writing stdlib packages to a directory the user doesn't own, or build go from source in a directory you don't own. I recommend the second option.

https://dave.cheney.net/2014/04/20/how-to-install-multiple-versions-of-go

@glycerine

This comment has been minimized.

glycerine commented Feb 7, 2017

Your makefile is calling go install, which fails because your user does not have permission to write to that directory.

wait.... go install is trying to write to $GOROOT, i.e. /usr/local/go? really? I thought go install would write only to $GOPATH, which is ~/go here.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 7, 2017

I agree that a simple go install for 1.8rc3 should not write to $GOROOT. Can you reproduce this without running make? What does go install -x print?

@davecheney

This comment has been minimized.

Contributor

davecheney commented Feb 7, 2017

wait.... go install is trying to write to $GOROOT, i.e. /usr/local/go? really? I thought go install would write only to $GOPATH, which is ~/go here.

If you set CGO_ENABLED=0 then the go tool has to rebuild all the standard lib packages, and because you used go install, it will try to write them back to their original locations. This is the same as cross compilation using the binary install of Go.

https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5, see comparison of go build vs go install.

@glycerine

This comment has been minimized.

glycerine commented Feb 7, 2017

What does go install -x print?

[jaten@biggie vhaline]$ export CGO_ENABLED=0                                                                                                    
[jaten@biggie vhaline]$ env|grep GO                                                                                                             
CGO_ENABLED=0                                                                                                                                   
GOPATH=/home/jaten/go                                                                                                                           
[jaten@biggie vhaline]$ go build                                                                                                                  
[jaten@biggie vhaline]$ go install -x                                                                                                           
WORK=/tmp/go-build958770575                                                                                                                     
mkdir -p $WORK/net/_obj/                                                                                                                        
mkdir -p $WORK/                                                                                                                                 
cd /usr/local/go/src/net                                                                                                                        
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/net.a -trimpath $WORK -p net -buildid 9b4851ad605915a6fa99ce0ea2840f1da29dd007 -D _/usr/loc\
al/go/src/net -I $WORK -pack ./addrselect.go ./cgo_stub.go ./conf.go ./dial.go ./dnsclient.go ./dnsclient_unix.go ./dnsconfig_unix.go ./dnsmsg.\
go ./fd_mutex.go ./fd_poll_runtime.go ./fd_posix.go ./fd_unix.go ./file.go ./file_unix.go ./hook.go ./hook_cloexec.go ./hook_unix.go ./hosts.go\
 ./interface.go ./interface_linux.go ./ip.go ./iprawsock.go ./iprawsock_posix.go ./ipsock.go ./ipsock_posix.go ./lookup.go ./lookup_unix.go ./m\
ac.go ./net.go ./nss.go ./parse.go ./pipe.go ./port.go ./port_unix.go ./sendfile_linux.go ./sock_cloexec.go ./sock_linux.go ./sock_posix.go ./s\
ockopt_linux.go ./sockopt_posix.go ./sockoptip_linux.go ./sockoptip_posix.go ./tcpsock.go ./tcpsock_posix.go ./tcpsockopt_posix.go ./tcpsockopt\
_unix.go ./udpsock.go ./udpsock_posix.go ./unixsock.go ./unixsock_posix.go ./writev_unix.go                                                     
mkdir -p /usr/local/go/pkg/linux_amd64/                                                                                                         
cp $WORK/net.a /usr/local/go/pkg/linux_amd64/net.a                                                                                              
go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied                                                                     
[jaten@biggie vhaline]$ cd ../cmd/vhaline/                                                                                                      
[jaten@biggie vhaline]$ go build                                                                                                                
[jaten@biggie vhaline]$ ldd ./vhaline                                                                                                           
    not a dynamic executable                                                                                                                    
[jaten@biggie vhaline]$   ##yeah! successfully built without cgo

Okay, then. So my immediately problem is solved. Thank you, Dave, for pointing the surprising difference between go build and go install under CGO_ENABLED=0.

@ianlancetaylor This is really pretty surprising behavior; I wouldn't have expected that CGO_ENABLED=0 would require a rebuild of the entire standard lib. If you want to leave the ticket open as a bug, that's okay with me. I'm also okay with closing it, at your discretion. Dave's cross-compilation post makes it clear for cross-compilation, but its not clear to me that setting CGO_ENABLED=0 is, or should be thought of, the same as cross-compiling.

Using go build vs go install

When cross compiling, you should use go build, not go install. This is the one of the few cases where go build is preferable to go install.

The reason for this is go install always caches compiled packages, .a files, into the pkg/ directory that matches the root of the source code.

For example, if you are building $GOPATH/src/github.com/lib/pq, then the compiled package will be >installed into $GOPATH/pkg/$GOOS_$GOARCH/github.com/lib/pq.a.

This logic also holds true for the standard library, which lives in /usr/local/go/src, so will be compiled to /usr/local/go/pkg/$GOOS_$GOARCH. This is a problem, because when cross compiling the go tool needs to rebuild the standard library for your target, but the binary distribution expects that /usr/local/go is not writeable.

Using go build rather that go install is the solution here, because go build builds, then throws away most of the result (rather than caching it for later), leaving you with the final binary in the current directory, which is most likely writeable by you.
-- from https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5

@rakyll

This comment has been minimized.

Member

rakyll commented Feb 7, 2017

Standard library builds (e.g. go install std) are going under GOROOT, wished it was somewhere under GOPATH at the cost of not being able to share prebuilt standard libraries (for various combinations of GOOS, GOARCH, CGO_ENABLED) between different GOPATHs.

@glycerine

This comment has been minimized.

glycerine commented Feb 7, 2017

(tongue in cheek...) perhaps $GOROOT==$GOPATH could mean "write my standard libs" under GOPATH.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 7, 2017

Setting CGO_ENABLED=0 does not require rebuilding the entire standard library, but it does require rebuilding the standard library packages that use cgo. Those packages are: net, os/user, crypto/x509.

That said, it does seem wrong that go install tries to install a standard library package in this case. If the package is not mentioned on the command line, it seems to me that it should be built but not installed.

@ianlancetaylor ianlancetaylor changed the title from linux/amd64 can't build with CGO_ENABLED=0: go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied to cmd/go: building with CGO_ENABLED=0 tries to install standard packages, which can fail Feb 7, 2017

@ianlancetaylor ianlancetaylor added this to the Go1.9 milestone Feb 7, 2017

@ianlancetaylor ianlancetaylor changed the title from cmd/go: building with CGO_ENABLED=0 tries to install standard packages, which can fail to cmd/go: `go install` with CGO_ENABLED=0 tries to install standard packages, which can fail Feb 7, 2017

@davecheney

This comment has been minimized.

Contributor

davecheney commented Feb 7, 2017

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 7, 2017

My feeling right now is that if you are using a release, then running go install without explicitly mentioning any standard library package should not attempt to install any standard library package. Clearly if you run go install std or go install net then it should install the standard library package (if it is out of date). But if you don't do that, I don't think go install should be trying to write files into GOROOT.

Happy to hear counter-arguments.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Feb 7, 2017

@petemoore

This comment has been minimized.

petemoore commented Mar 3, 2017

Note, in our case, this occurred with go version go1.7.3 linux/amd64 - see taskcluster/taskcluster-worker#190 (comment)

@stapelberg

This comment has been minimized.

Contributor

stapelberg commented Mar 18, 2017

Strongly agree with @davecheney.

In https://gokrazy.github.io/, a hurdle which multiple users have stumbled over is that Go installations from installers and distribution packages will fail cross-compilation with a “permission denied” error in the standard library.

I.e., people run something like GOOS=linux GOARCH=arm64 go get -u github.com/gokrazy/hello and the command fails with go install runtime/internal/sys: mkdir /usr/lib/go-1.7/pkg/linux_arm64: permission denied. The error message is from Debian, but the problem also affects macOS.

I think many of us are blind to that issue because we install Go from source. End users typically don’t.

@rsc

This comment has been minimized.

Contributor

rsc commented Jun 22, 2017

Work on better caching will fix this. I have a prototype. Not for Go 1.9 but probably for Go 1.10.

@gopherbot

This comment has been minimized.

gopherbot commented Nov 3, 2017

Change https://golang.org/cl/75850 mentions this issue: cmd/go: do not install dependencies during "go install"

ethanfrey added a commit to tendermint/tendermint that referenced this issue Dec 8, 2017

Remove CGO_ENABLED=0 from make install
It was writing to stdlib packages net, x/crypto, etc. and failing when it didn't have write access to them (which it often shouldn't)

Fixes issue #941

Explained in golang issue: golang/go#18981 (fixed in develop/1.10)

ethanfrey added a commit to tendermint/tendermint that referenced this issue Dec 8, 2017

Remove CGO_ENABLED=0 from make install
It was writing to stdlib packages net, x/crypto, etc. and failing when it didn't have write access to them (which it often shouldn't)

Fixes issue #941

Explained in golang issue: golang/go#18981 (fixed in develop/1.10)

slackpad added a commit to hashicorp/envconsul that referenced this issue Jan 22, 2018

Removes disabling CGO for dev builds.
This works around golang/go#18981 until that
fix makes it into an official release.

wking added a commit to wking/bootkube that referenced this issue Apr 3, 2018

Documentation/development: Bump minimum Go version to 1.10
This gets us a fixed 'CGO_ENABLED=0 go install' [1,2] and avoids 1.8
(which is no longer supported upstream now that 1.10 has been released
[3]).

I've also dropped the version information from the build comment (it's
mentioned just before in the "Requirements" section) to DRY things up
a bit.

[1]: golang/go#18981
[2]: golang/go@8f70e1f
[3]: https://golang.org/doc/devel/release.html#policy

wking added a commit to wking/bootkube that referenced this issue Apr 3, 2018

Makefile: Use 'go install ...' for the install target
GOBIN (documented in [1]) defaults to DIR/bin, but it's a configurable
variable in its own right.  The old logic was just looking at GOPATH,
though, and not respecting GOBIN.

This commit updates our install target to lean on Go's build caching
(instead of using Make's dependency trees) to ensure a fresh-enough
build lands in the appropriate directory.  This approach relies on Go
1.10+ to avoid [2], but we've required Go 1.10+ since b859ebf
(Documentation/development: Bump minimum Go version to 1.10,
2018-04-03, kubernetes-incubator#955).

For the build documentation, I've switched to 'go env GOPATH' to get
the (platform-specific [1]) default value when the environment
variable is not set.  And I've used cut [3] (instead of the awk [4]
the Makefile used to use) to pull out the first component of GOPATH.
Both are in POSIX, but cut is a simpler tool for this particular
problem.

[1]: https://golang.org/cmd/go/#hdr-GOPATH_environment_variable
[2]: golang/go#18981
[3]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cut.html
[4]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html

rphillips added a commit to kubernetes-incubator/bootkube that referenced this issue Apr 3, 2018

Makefile: Use 'go install ...' for the install target (#949)
GOBIN (documented in [1]) defaults to DIR/bin, but it's a configurable
variable in its own right.  The old logic was just looking at GOPATH,
though, and not respecting GOBIN.

This commit updates our install target to lean on Go's build caching
(instead of using Make's dependency trees) to ensure a fresh-enough
build lands in the appropriate directory.  This approach relies on Go
1.10+ to avoid [2], but we've required Go 1.10+ since b859ebf
(Documentation/development: Bump minimum Go version to 1.10,
2018-04-03, #955).

For the build documentation, I've switched to 'go env GOPATH' to get
the (platform-specific [1]) default value when the environment
variable is not set.  And I've used cut [3] (instead of the awk [4]
the Makefile used to use) to pull out the first component of GOPATH.
Both are in POSIX, but cut is a simpler tool for this particular
problem.

[1]: https://golang.org/cmd/go/#hdr-GOPATH_environment_variable
[2]: golang/go#18981
[3]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cut.html
[4]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html

@golang golang locked and limited conversation to collaborators Nov 3, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.