Repository navigation
Closed
Labels
FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.ToolSpeedcompiler/runtimeIssues related to the Go compiler and/or runtime.Issues related to the Go compiler and/or runtime.
Milestone
Description
Go version
go version go1.21.5 darwin/amd64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/thomaspeugeot/Library/Caches/go-build'
GOENV='/Users/thomaspeugeot/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/thomaspeugeot/go/pkg/mod'
GONOPROXY='github.com/thomaspeugeot,github.com/fullstack-lang'
GONOSUMDB='github.com/thomaspeugeot,github.com/fullstack-lang'
GOOS='darwin'
GOPATH='/Users/thomaspeugeot/go'
GOPRIVATE='github.com/thomaspeugeot,github.com/fullstack-lang'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/thomaspeugeot/go/src/github.com/thomaspeugeot/thelongbuild/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/_r/bxjvdzc17mb6p56qc7nms1r80000gn/T/go-build65103854=/tmp/go-build -gno-record-gcc-switches -fno-common'What did you do?
below are the steps to reproduce the timing result
on macos
rm -rf thelongbuild
git clone https://github.com/thomaspeugeot/thelongbuild.git
cd thelongbuild/go/tree
go clean -cache
time go build -v
sed -i '' 's/gongtree_buttons/gongtree_buttons_new/g' tree.go
time go build -v
cd ../../..
the sed command is to trick the compiler to dirty its cache.
note for other plateforms, the sed command is different, it is sed -i 's/gongtree_buttons/gongtree_buttons_new/g' tree.go
What did you see happen?
go build -v 46.99s user 2.44s system 371% cpu 13.320 total
What did you expect to see?
less than a few seconds
Metadata
Metadata
Assignees
Labels
FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.ToolSpeedcompiler/runtimeIssues related to the Go compiler and/or runtime.Issues related to the Go compiler and/or runtime.
Type
Projects
- StatusShow more project fieldsDone
Activity
seankhliao commentedon Jan 13, 2024
i believe this is working as expected as the standard library is not precompiled as part of the distribution but part of the user build cache.
thomaspeugeot commentedon Jan 13, 2024
cespare commentedon Jan 13, 2024
Reopening. Thanks for the easy reproducer @thomaspeugeot.
seankhliao commentedon Jan 13, 2024
is there antivirus or other endpoint security software on the machine?
and where does it spend its time if you use
go build -xthomaspeugeot commentedon Jan 13, 2024
No
cespare commentedon Jan 13, 2024
I can reproduce locally on my Linux machine.
The time is spent actually running
compile(i.e. gc).I looked at a CPU profile and it's not obvious what's wrong, but there's a lot of GC. A heap profile shows a bunch of SSA-related allocations but I don't have enough experience debugging the compiler to know what's wrong based on that. The heap grows to 2.5 GB when compiling this which seems excessive? If I had to guess, something about this code causes the compiler to generate a rather large/inefficient internal representation of something.
[-]long buid time (45 seconds) for a small package (1500 source lines of code)[/-][+]cmd/compile: long buid time (45 seconds) for a small package (1500 source lines of code)[/+]cespare commentedon Jan 13, 2024
BTW, it doesn't take 45s for me, but about 10s. I'm on a beefy desktop machine, though. I compared to a few other Go programs of similar size and compiling them (when deps are cached) takes roughly 200-500ms for me, so it's still >1 order of magnitude slower than I'd expect.
[-]cmd/compile: long buid time (45 seconds) for a small package (1500 source lines of code)[/-][+]cmd/compile: long build time (45 seconds) for a small package (1500 source lines of code)[/+]seankhliao commentedon Jan 13, 2024
maybe it's the generics like with #51957 ? though there's less code here
mauri870 commentedon Jan 14, 2024
A brief look at cpu and mem profiles I can see that the compiler is taking a beating to compile all the switch statements over generics in the code.
Edit: a lot of time spent in ssa.Compile, cse, deadcode and prove passes
thomaspeugeot commentedon Jul 15, 2024
@mdempsky , i prepared a branch "simplification-of-the-code" that is a stripped down version (48 lines instead of 1500 lines)
I noticed 2 things:
1/ With the following line of code, one goes from below 1 second to above 43 seconds
2/ 43 seconds is also the time it takes to build the
probepackageBy adding the line in the "tree" package", it seems that go rebuilds the whole "probe" package.
egonelbre commentedon Jul 15, 2024
Some alternatives until things can be improved in the compiler. These may not fit your application, but it might fit someone else facing similar problems.
You might get help from not specializing on the funcs that mainly use
switch any(x).(type) {. An example:If you still want the type safety of the call, you can use:
If possible then you might be better of generating things as methods and using interfaces instead. This should allow the compiler to optimize things easier.
Similarly, there seem to be places where you can replace things with direct calls. For example:
And I wouldn't be surprised if using interfaces / reflection more in the implementation you would get even better result.
thomaspeugeot commentedon Jul 15, 2024
Hi @egonelbre , thanks for the pieces of advice. I will try to apply them. Most of the time, i rather use methods over function. However, most of this code is generic and using a method can be tricky (you can use a method on a generic struct). This whole code is a proof of concept that was started before the generics arrived.
BTW, I just found out the source of 50% of the build slowness. It happens that the function
FillUpNamedFormFromGongstruct[T models.Gongstruct](instance *T, ...in theprobepackage that is called from thetreepackage is using a generic functionGetReverseFieldOwnerName[T models.Gongstruct(...]in another package namedorm. This generic function in theormpackage is a generated file with 5754 lines of code with a lot of dead code. By removing the dead code in this function, one goes down to 626 lines. Then, the compilation time of thetreepackage goes to 20 seconds, a 50% saving (You can check the result in the branch "simplified-orm").As a conclusion, I wonder if the
go buildslowness is still a problem. Are we not in a fringe use case with a lot of generated code ?egonelbre commentedon Jul 16, 2024
It does seem to me on the fringe; however these situations can make it easier to discover general optimizations that help with regular codebases.