cmd/compile: compilation of a few large maps blows up compilers memory usage

[gustafj]

What version of Go are you using (go version)?

1.9.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build186394948=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compile (go tool compile, go build, go run, etc)

See gist https://gist.github.com/gustafj/aa3c2e7ba4b3dc2b4701282e73d81646 for a sample that generates a sample program that ends up using ~1GB of ram.

• $ go run main.go
• $ $(go tool -n compile) -memprofile m.prof -goversion go1.9.2 /tmp/go_compile_test.go
• $ go tool pprof -top -alloc_space go tool -n compile m.prof | head -20

File: compile
Type: alloc_space
Showing nodes accounting for 1215.06MB, 94.58% of 1284.72MB total
Dropped 125 nodes (cum <= 6.42MB)
      flat  flat%   sum%        cum   cum%
  852.05MB 66.32% 66.32%   852.05MB 66.32%  cmd/compile/internal/ssa.(*regAllocState).computeLive /usr/lib/go/src/cmd/compile/internal/ssa/regalloc.go
   59.02MB  4.59% 70.92%    59.02MB  4.59%  cmd/compile/internal/ssa.(*Func).newValue /usr/lib/go/src/cmd/compile/internal/ssa/func.go
   50.51MB  3.93% 74.85%    50.51MB  3.93%  cmd/compile/internal/gc.(*Progs).Prog /usr/lib/go/src/cmd/compile/internal/gc/gsubr.go
   35.74MB  2.78% 77.63%    35.74MB  2.78%  cmd/compile/internal/ssa.(*Func).setHome /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
      31MB  2.41% 80.04%       31MB  2.41%  cmd/compile/internal/gc.nodl /usr/lib/go/src/cmd/compile/internal/gc/subr.go
   20.21MB  1.57% 81.62%   875.33MB 68.13%  cmd/compile/internal/ssa.(*regAllocState).init /usr/lib/go/src/cmd/compile/internal/ssa/regalloc.go
   15.54MB  1.21% 82.82%    15.54MB  1.21%  cmd/compile/internal/ssa.(*stackAllocState).stackalloc /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
   15.28MB  1.19% 84.01%    15.83MB  1.23%  cmd/compile/internal/ssa.cse /usr/lib/go/src/cmd/compile/internal/ssa/cse.go
   15.26MB  1.19% 85.20%    15.26MB  1.19%  cmd/compile/internal/ssa.storeOrder /usr/lib/go/src/cmd/compile/internal/ssa/schedule.go
   13.17MB  1.03% 86.23%    13.17MB  1.03%  cmd/compile/internal/ssa.schedule /usr/lib/go/src/cmd/compile/internal/ssa/schedule.go
   11.50MB   0.9% 87.12%    11.50MB   0.9%  cmd/compile/internal/gc.newnamel /usr/lib/go/src/cmd/compile/internal/gc/subr.go
    9.10MB  0.71% 87.83%     9.10MB  0.71%  cmd/compile/internal/ssa.(*Func).NewBlock /usr/lib/go/src/cmd/compile/internal/ssa/func.go
    8.72MB  0.68% 88.51%     8.72MB  0.68%  cmd/compile/internal/ssa.(*stackAllocState).buildInterferenceGraph /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
    8.72MB  0.68% 89.19%    20.39MB  1.59%  cmd/compile/internal/ssa.(*stackAllocState).init /usr/lib/go/src/cmd/compile/internal/ssa/stackalloc.go
    8.53MB  0.66% 89.85%     8.53MB  0.66%  cmd/internal/obj/x86.(*AsmBuf).doasm /usr/lib/go/src/cmd/internal/obj/data.go

What did you expect to see?

Fast compilation and a memory usage of max a couple of megs.

What did you see instead?

Slow compilation and a memory usage of almost a gig.
(With a little larger map the end result will be SIGKILL due to out of memory, which for me is ~ 30GB)

[ALTree]

I think you're hitting #21713. I tried your reproducer on tip and the memory usage seems to be of just a few MB. Can you try with tip (to be 1.10) and confirm that the issue is indeed already fixed?

[gustafj]

Thanks for the fast response.
The issue is indeed fixed in tip.
Both my test and the original code that triggered the issue compiles with "normal" memory footprint (2.5MB / 37MB) when using tip.

[ALTree]

Nice. Since this is fixed on tip (and the fix won't be backported to earlier releases) there's nothing left to do, so I'm closing this issue. The fix will be included in the next go release (1.10).

[gustafj]

Ok, just for reference (since it is discussed in the referenced issue), the code that triggers this behaviour is not generated.
It is a couple quite large lookup tables which are manually created.