cmd/compile: Go compiler eats all available system memory

[zlepper]

What version of Go are you using (go version)?

go version go1.8 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\development\goprojects
set GORACE=
set GOROOT=D:\Go
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\hanse\AppData\Local\Temp\go-build350759391=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Tried to compile to following using go build .
gocompiler.zip

I suspect it's the autogenerated hosting.go file that causes things to explode due to some quite huge string (It's an embedded angular application)

What did you expect to see?

A build program

What did you see instead?

All my memory being eaten.

I have to kill the compiler as it after a couple of seconds uses more than 10GB.
If i just let it run it eats all the memory available on my machine, and the machine has to be reset using the hardware switch.

[bradfitz]

Many of these sorts of bugs were fixed recently.

Could you try Go 1.9beta1? https://golang.org/dl/#unstable

Note that on Windows, the MSI installer will uninstall your old version (that is #16305), so you'll want to use the zip version for testing probably.

[zlepper]

Attempting to run the build command just gives me the following:

C:\Users\hanse\Desktop\gocompiler>"D:\Downloads\go1.9beta1.windows-amd64\go\bin\go.exe" build .
# _/C_/Users/hanse/Desktop/gocompiler
flag provided but not defined: -goversion
usage: compile [options] file.go...

[bradfitz]

That means your environment is messed up. You're running a mix of Go versions. Unset or fix your GOROOT.

[zlepper]

Fixed my env, attempted to compile with 1.9beta1: Memory consumption still explodes.

go version:
go version go1.9beta1 windows/amd64

go env:
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\development\goprojects
set GORACE=
set GOROOT=D:\Downloads\go1.9beta1.windows-amd64\go
set GOTOOLDIR=D:\Downloads\go1.9beta1.windows-amd64\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\hanse\AppData\Local\Temp\go-build564923191=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

[josharian]

I strongly suspect that this is a duplicate of #16394. If so, it is a real bug that needs fixing, but there is a good workaround available: Avoid string concatenation of many/large constant strings in generated code. Instead, use strconv.Quote to escape the string and emit it verbatim.

[zlepper]

Indeed, using strconv.Quote fixed gets arround the issue. (And for what i need, it's much more elegant too :) )

The only difference i see between this issue, and the one @josharian linked, is that the go compiler doesn't actually crash here, it makes the system run out of memory (At least on my system running with 16GB spare)

[josharian]

Given enough memory, the compiler would crash. :) I'm going to call this a dup, since the underlying cause is the same. Thanks for checking.

[zlepper]

Shouldn't the compiler give up before it kills the system it's running on? Having to hardware reset the computer is not exactly optimal?

[bradfitz]

Shouldn't the compiler give up before it kills the system it's running on? Having to hardware reset the computer is not exactly optimal?

@zlepper, Josh isn't denying there's a problem. He closed it as a dup of #16394.

Go should use as little memory as possible and not go into runaway memory situations like this, but on the other hand, it's not Go's job to tiptoe around your operating system. If your operating system doesn't have the memory, it's the OS's job to kill the process, not wedge itself and require a physical hardware reset.

But this will be fixed. Subscribe to #16394.

[zlepper]

That is true, had not considered that angle. Thank you.