Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: compiler hangs without optimizations #29144

csnitker opened this issue Dec 7, 2018 · 10 comments

cmd/compile: compiler hangs without optimizations #29144

csnitker opened this issue Dec 7, 2018 · 10 comments
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed


Copy link

csnitker commented Dec 7, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 windows/amd64

Does this issue reproduce with the latest release?

Yes, reproduced using go 1.11, 1.11.2 and 1.10

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\csnitker\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\csnitker\go
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set GOMOD=C:\Users\csnitker\go\src\\RegistrarServices\whois\go.mod
set CGO_CFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\csnitker\AppData\Local\Temp\go-build373743682=/tmp/go-build -gno-record-gcc-switches

What did you do?

Run go build -gcflags=all="-N -l" -x main.go or dlv debug main.go on the following:

package main

import ""

func main() {
    zone := zonedb.PublicZone("")
    if zone != nil {

What did you expect to see?

A successful build.

What did you see instead?

The compiler hangs and consumes large amounts of memory.


The command from the process above shows:

C:\go1.11\pkg\tool\windows_amd64\compile.exe -o C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\_pkg_.a -trimpath C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002 -N -l -p -complete -buildid mw6Bk1kcUZg16C0mq6E3/mw6Bk1kcUZg16C0mq6E3 -goversion go1.11 -D "" -importcfg C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\importcfg -pack -c=4 C:\Users\csnitker\go\pkg\mod\\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zone.go C:\Users\csnitker\go\pkg\mod\\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zones.go

It's also interesting that go build -gcflags="-N -l" main.go completes successfully and works using dlv exec main.exe but does not with the commands above. I'm not entirely sure if this is a go problem or an issue with the library (or delve) but figured I would start here since an optimized build completes without issue.

@ianlancetaylor ianlancetaylor changed the title compile hangs without optimizations cmd/compile: compiler hangs without optimizations Dec 7, 2018
Copy link

The file has a fairly large composite literal. I suspect that the compiler isn't hanging as such, it's just taking a lot of memory, perhaps causing your system to thrash.

CC @randall77

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 7, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Dec 7, 2018
Copy link

Although actually it doesn't take very long to build that package on my laptop.

Copy link

It's the difference between -gcflags=all="-N -l" and -gcflags="-N -l". The former applies the optimizations-off flags to all packages (including the zonedb package) and the latter to only the top-level package and not its imports. The latter will use an optimized version of the zonedb package which compiles somewhat quickly.

It's the -N, removing -l doesn't change anything.

Copy link

Looks like it is spending most of its time (and space) in live variable analysis in the register allocator.
I'm not sure there's a whole lot we can do about that, unfortunately.
Maybe if there was a key phase that, if enabled even with -N, would make the function easier for regalloc to handle.

Copy link

That's impressive. With -gcflags=all=-N I let the compiler run for ten minutes before killing it, at which point the virtual memory size was over 15G.

Why do we need live variable analysis with -N? In a big function can we just dump everything on the stack?

Copy link

We could implement a second, simpler register allocator for -N, sure. I was hoping to avoid that.
I'm not sure how much we could share between the two allocators. The live variable analysis is pretty integral to the current design, I don't think we could just say "everything is live" and have it work.

@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Copy link

icamys commented Nov 29, 2019

Hi all! The bug also can be reproduced with go version go1.13 linux/amd64

Copy link

Another confirmation. This gets triggered when running go code in intellij on debug mode, if the zonedb package is added to the code, as Intellij adds those gcflags by default.

Version: go version go1.13.3 linux/amd64

Copy link

If someone wants to dig into this, it might be useful to know whether there is one key optimization omitted by -N that makes the difference. If so, perhaps (say) there might be a way to do a subset of that optimization even with -N to keep sizes reasonable.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
Copy link

ivanmemruk commented Apr 27, 2023

This reproduces on 1.20.3 on darwin/arm64. Specifically, because of this handy module which contains some large literals as well, for example the one in (beware, it kinda hangs Chrome for a bit as well).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Status: Triage Backlog

No branches or pull requests

10 participants