Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go build' does not cache linked executables #31629

Open
dan-lind opened this issue Apr 23, 2019 · 12 comments

Comments

@dan-lind
Copy link

commented Apr 23, 2019

What version of Go are you using (go version)?

$ go version
 go1.12.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ko1dli/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ko1dli/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/mnt/c/Users/ko1dli/git/serverless-alias-search/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build698955985=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ time go build ./...
go build ./...  55.27s user 23.34s system 278% cpu 28.186 total
$ time go build ./...
go build ./...  57.38s user 22.05s system 275% cpu 28.872 total
$ time go install ./...
go install ./...  56.70s user 24.88s system 274% cpu 29.669 total
$ time go install ./...
go install ./...  0.70s user 2.67s system 168% cpu 2.003 total

What did you expect to see?

I expected go build to do an incremental build, but it seems like it recompiles everything every time I build. See go install for comparison

What did you see instead?

I expected the second go build to be incremental, rather than rebuilding everything again.

@katiehockman katiehockman changed the title Incremental builds not working with go build, only with go install cmd/go: go build does not build incrementally like go install Apr 29, 2019

@katiehockman

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Apr 30, 2019

go build should be building incrementally.

Can you give us more info to reproduce this? Specifically, we'd be looking for a test repository that reproduces this problem and output of two consecutive runs of go build -x ./... that shows the same commands being executed.

@dan-lind

This comment has been minimized.

Copy link
Author

commented May 2, 2019

I've added the output of three consecutive time go build -x ./... after first doing go clean
The first one obviously takes longer because of some downloads, but the seconds and third ones are identical from what I can see. Is this helpful by itself?

I will look into providing a test repo, but I reckon it might take some time since the code is not open source.

build1.txt
build2.txt
build3.txt

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented May 10, 2019

It looks like all the actions in the second and third logs are link actions. That makes sense I guess. Executables never get written to the cache, but when executables are installed (with go install, go get, go build -i), the link may be skipped for target executables that are up to date.

Executables were originally not written to the cache because before 1.12, caching was optional, and there was nowhere to put them outside of the cache if we weren't installing them.

Now that the cache is mandatory, we could store executables in the cache fairly easily. This may increase cache size significantly though, and it doesn't seem like there's much benefit.

@dan-lind

This comment has been minimized.

Copy link
Author

commented May 11, 2019

@jayconrod You say that executables never get written to the cache, and mention go build -i which makes me a bit puzzled. The Go 1.10 release notes says

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.

As I understand it they should be written to the cache, although a separate one, even without using the -i option?

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented May 13, 2019

I think that snippet is referring to packages (.a files) only. I don't see mention of binaries.

When building with go install, we do treat installed binaries as part of the cache, even though they aren't in the cache directory. That's why you saw the speedup earlier on the second go install command.

@jeffsaremi

This comment has been minimized.

Copy link

commented Sep 15, 2019

I'm experiencing the exact same thing. Never been able to do any incremental builds. Go build or even Go install will build the whole thing every time

go version go1.9.3 linux/amd64

I can give you the complete project to investigate once you let me know how I can upload files for you

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 16, 2019

@jeffsaremi, per the release policy Go 1.9 has not been supported since 2018-08-24 (when Go 1.11 was released). Please upgrade to a supported version (1.12 or 1.13) and see whether you can still reproduce the issue.

@jeffsaremi

This comment has been minimized.

Copy link

commented Sep 16, 2019

Thank you. I shall do as instructed and report back

@bcmills bcmills changed the title cmd/go: go build does not build incrementally like go install cmd/go: 'go build' does not cache linked executables Sep 16, 2019

@bcmills bcmills added the ToolSpeed label Sep 16, 2019

@jeffsaremi

This comment has been minimized.

Copy link

commented Sep 16, 2019

As per @bcmills recommendation, I upgraded go and the problem is solved. go build now does incremental build with beautifully
thanks

go version go1.13 linux/amd64

@dan-lind

This comment has been minimized.

Copy link
Author

commented Sep 18, 2019

@jeffsaremi I guess you did go install then?
With go build I'm still seeing the same behaviour as before with 1.13, which is what I expect since this issue is still open :-)

@jeffsaremi

This comment has been minimized.

Copy link

commented Sep 18, 2019

No I did a plain go build. It was smart enough not to rebuild the final binary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.