New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

Open
yunabe opened this Issue Feb 22, 2018 · 4 comments

Comments

Projects
None yet
7 participants
@yunabe

yunabe commented Feb 22, 2018

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yunabe/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/yunabe/local/gocode"
GORACE=""
GOROOT="/home/yunabe/go1.10"
GOTMPDIR=""
GOTOOLDIR="/home/yunabe/go1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build331277736=/tmp/go-build -gno-record-gcc-switches"

What did you do?

cd $TMPDIR
go get -d golang.org/x/net/context
go install -buildmode=shared -linkshared -pkgdir `pwd`/tmppkg -linkshared std
go install -x -buildmode=shared -pkgdir `pwd`/tmppkg -linkshared golang.org/x/net/context

What did you expect to see?

The second go install command compiles context.a and link libgolang.org-x-net-context.so. It takes less than 1 sec to finish.

What did you see instead?

From go1.10, the second go install run link command to create libstd.so though libstd.so already exists in -pkgdir.
It makes the second go install command 3-4 times slower (0.77sec in go1.9.4 vs 2.8sec in go1.10)

I debugged the behavior of go tools and it seems like b.useCache in linkShared returns false to libstd.so target (https://github.com/golang/go/blob/master/src/cmd/go/internal/work/exec.go#L1032).

@ianlancetaylor ianlancetaylor changed the title from go install -linkshared invokes link commands for existing .so files since go1.10 to cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 Feb 22, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.10.1 milestone Feb 22, 2018

@yunabe yunabe closed this Feb 22, 2018

@yunabe yunabe reopened this Feb 22, 2018

@andybons andybons modified the milestones: Go1.10.1, Go1.10.2 Mar 27, 2018

@AlphaWong AlphaWong referenced this issue Apr 22, 2018

Closed

Slow startup #23

@FiloSottile FiloSottile modified the milestones: Go1.10.2, Go1.11 Apr 24, 2018

@FiloSottile

This comment has been minimized.

Member

FiloSottile commented Apr 24, 2018

@gopherbot please open a backport tracking issue, as this is a 1.10 regression, although just a performance one.

@gopherbot

This comment has been minimized.

gopherbot commented Apr 24, 2018

Backport issue(s) opened: #25044 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@yunabe yunabe referenced this issue Apr 25, 2018

Closed

go1.10 support #32

@Foxboron

This comment has been minimized.

Foxboron commented May 22, 2018

This isn't actually just a performance issue i believe. Currently this prevents users from running -buildmode=shared -linkshared as we don't have write privileges inn $GOROOT, and it attempts to write towards several files there.

With go 1.10

==> Starting build()...
WORK=/tmp/go-build138655936
rm -r $WORK/b003/
rm -r $WORK/b011/
rm -r $WORK/b007/
rm -r $WORK/b022/
mkdir -p $WORK/b010/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/
cp /home/fox/.cache/go-build/20/203799d24b5628cee748f251b4361e400b7392a9e65f21f6345ecb0ac1e99ca2-d /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a
rm -r $WORK/b010/
go install runtime/internal/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a: permission denied
rm -r $WORK/b023/
rm -r $WORK/b037/
mkdir -p $WORK/b012/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/sync/
cp /home/fox/.cache/go-build/59/59efc6e118c62c218e3e59d3e843f2e6895024fd8cfe6cba4d14ad046db4f7c4-d /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a
rm -r $WORK/b047/
rm -r $WORK/b012/
go install sync/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a: permission denied
rm -r $WORK/b048/
rm -r $WORK/b052/
mkdir -p $WORK/b014/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/internal/
cp /home/fox/.cache/go-build/5c/5cad130c59015083c83300f528455ed005db9a7862397945d4740e5e35e8f044-d /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a
rm -r $WORK/b014/
go install internal/cpu: open /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a: permission denied
mkdir -p $WORK/b053/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/
cp /home/fox/.cache/go-build/ae/aee318ece67832c036cdbc8be92aef902422e29e53942e16e2935a2dd7a7f0e7-d /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a
rm -r $WORK/b053/
rm -r $WORK/b077/
go install crypto/internal/cipherhw: open /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a: permission denied
rm -r $WORK/b079/
rm -r $WORK/b094/
rm -r $WORK/b100/
mkdir -p $WORK/b085/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/
cp /home/fox/.cache/go-build/27/278eecfb4ee5a9a34edbfc24a41c030a7f28e489c9a1edcc610606a7684baf40-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a
rm -r $WORK/b085/
mkdir -p $WORK/b084/
rm -r $WORK/b146/
cp /home/fox/.cache/go-build/a8/a883a42b6f584a222b93fd8c5df99499fd4dba6efcf90a2eb564ea90cdad6611-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a
rm -r $WORK/b084/
go install vendor/golang_org/x/crypto/poly1305: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a: permission denied
rm -r $WORK/b154/
rm -r $WORK/b155/
rm -r $WORK/b147/
rm -r $WORK/b156/
go install vendor/golang_org/x/crypto/curve25519: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a: permission denied
rm -r $WORK/b187/
rm -r $WORK/b186/
rm -r $WORK/b025/
rm -r $WORK/b180/

Meanwhile go 1.9.4

==> Starting build()...
WORK=/tmp/go-build073266895
golang.org/x/net/context
mkdir -p $WORK/golang.org/x/net/context/_obj/
mkdir -p $WORK/golang.org/x/net/
cd /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context
/usr/lib/go/pkg/tool/linux_amd64/compile -o $WORK/golang.org/x/net/context.a -trimpath $WORK -dynlink -goversion go1.9.4 -p golang.org/x/net/context -complete -installsuffix dynlink -buildid 3d063c354b94a000bba5463988c84e19a35173d6 -D _/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context -I $WORK -pack ./context.go ./go17.go ./go19.go
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/
cp $WORK/golang.org/x/net/context.a /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
cd .
/usr/lib/go/pkg/tool/linux_amd64/link -o $WORK/libgolang.org-x-net-context.so -L $WORK -L /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink -installsuffix dynlink -buildmode=shared -linkshared -w -extld=gcc golang.org/x/net/context=/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
echo 'libgolang.org-x-net-context.so' > /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.shlibname # internal
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/
cp $WORK/libgolang.org-x-net-context.so /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/libgolang.org-x-net-context.so
@rsc

This comment has been minimized.

Contributor

rsc commented Aug 10, 2018

Sorry but not going to get to this for Go 1.11. Linkshared is admittedly a mess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment