Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: compiler hangs without optimizations #29144

Open
csnitker opened this issue Dec 7, 2018 · 6 comments

Comments

@csnitker
Copy link

commented Dec 7, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 windows/amd64

Does this issue reproduce with the latest release?

Yes, reproduced using go 1.11, 1.11.2 and 1.10

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\csnitker\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\csnitker\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\csnitker\go\src\go.registrar.int.godaddy.com\RegistrarServices\whois\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\csnitker\AppData\Local\Temp\go-build373743682=/tmp/go-build -gno-record-gcc-switches

What did you do?

Run go build -gcflags=all="-N -l" -x main.go or dlv debug main.go on the following:

package main

import "github.com/zonedb/zonedb"

func main() {
    zone := zonedb.PublicZone("godaddy.com")
    if zone != nil {
        println(zone)
    }
}

What did you expect to see?

A successful build.

What did you see instead?

The compiler hangs and consumes large amounts of memory.

image

The command from the process above shows:

C:\go1.11\pkg\tool\windows_amd64\compile.exe -o C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\_pkg_.a -trimpath C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002 -N -l -p github.com/zonedb/zonedb -complete -buildid mw6Bk1kcUZg16C0mq6E3/mw6Bk1kcUZg16C0mq6E3 -goversion go1.11 -D "" -importcfg C:\Users\csnitker\AppData\Local\Temp\go-build516058638\b002\importcfg -pack -c=4 C:\Users\csnitker\go\pkg\mod\github.com\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zone.go C:\Users\csnitker\go\pkg\mod\github.com\zonedb\zonedb@v0.0.0-20181207082023-cee273e3e759\zones.go

It's also interesting that go build -gcflags="-N -l" main.go completes successfully and works using dlv exec main.exe but does not with the commands above. I'm not entirely sure if this is a go problem or an issue with the library (or delve) but figured I would start here since an optimized build completes without issue.

@ianlancetaylor ianlancetaylor changed the title compile hangs without optimizations cmd/compile: compiler hangs without optimizations Dec 7, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

The file github.com/zonedb/zonedb/zones.go has a fairly large composite literal. I suspect that the compiler isn't hanging as such, it's just taking a lot of memory, perhaps causing your system to thrash.

CC @randall77

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Dec 7, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

Although actually it doesn't take very long to build that package on my laptop.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

It's the difference between -gcflags=all="-N -l" and -gcflags="-N -l". The former applies the optimizations-off flags to all packages (including the zonedb package) and the latter to only the top-level package and not its imports. The latter will use an optimized version of the zonedb package which compiles somewhat quickly.

It's the -N, removing -l doesn't change anything.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

Looks like it is spending most of its time (and space) in live variable analysis in the register allocator.
I'm not sure there's a whole lot we can do about that, unfortunately.
Maybe if there was a key phase that, if enabled even with -N, would make the function easier for regalloc to handle.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

That's impressive. With -gcflags=all=-N I let the compiler run for ten minutes before killing it, at which point the virtual memory size was over 15G.

Why do we need live variable analysis with -N? In a big function can we just dump everything on the stack?

@randall77

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2018

We could implement a second, simpler register allocator for -N, sure. I was hoping to avoid that.
I'm not sure how much we could share between the two allocators. The live variable analysis is pretty integral to the current design, I don't think we could just say "everything is live" and have it work.

@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.