Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: memory usage above 1.5 GB #33421

Closed
rillig opened this issue Aug 1, 2019 · 5 comments
Closed

cmd/compile: memory usage above 1.5 GB #33421

rillig opened this issue Aug 1, 2019 · 5 comments
Milestone

Comments

@rillig
Copy link
Contributor

@rillig rillig commented Aug 1, 2019

What version of Go are you using (go version)?

go 1.12.7 on NetBSD/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rillig/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="netbsd"
GOOS="netbsd"
GOPATH="/home/rillig/go"
GOPROXY=""
GORACE=""
GOROOT="/home/rillig/minipkg/go112"
GOTMPDIR=""
GOTOOLDIR="/home/rillig/minipkg/go112/pkg/tool/netbsd_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build435520277=/tmp/go-build -gno-record-gcc-switches"

What did you do?

bootstrap go1.12.7 using go14, via pkgsrc, the NetBSD package management framework

cd /usr/pkgsrc/lang/go112
bmake update

What did you expect to see?

Compiling go code takes less than 100 MB (ideally). 500 MB would already seem too high but still acceptable. But 1.7 GB is really too much.

What did you see instead?

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
 1160 rillig    78    0  1262M  675M RUN/0      0:31 85.50% 65.48% compile

The memory usage even went up to 1735 MB.

@ianlancetaylor ianlancetaylor added this to the Go1.14 milestone Aug 2, 2019
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 2, 2019

Can you find out what is being compiled when the compiler uses that much memory?

@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Aug 2, 2019

You wrote that what you're doing is:

bootstrapping go1.12.7 using go14

so I assume you're measuring the memory consumption of the old go1.4 toolchain while it builds the newest toolchain. That old 1.4 toolchain was written in C and it is kept around just for bootstrapping purposes, and it's basically frozen. It won't be optimized to make it faster or consume less memory.

Your screenshot shows the process using 675M of Resident Memory (RES), which is what you care about, so the title of your issue doesn't match your report, as it doesn't appear the toolchain is using 1.7GB of memory.

The newest compilers are written in Go and they also requires a fair amount of memory when bootstrapping master, anyway. You won't be able to compile the Go toolchain using 100 or 500 MB. AFAIK on 64-bit machines you'll need close to 800MB of memory per core. If you restrict make.bash to use one core, you should be able to build the latest toolchain using around 1GB of memory.

@rillig

This comment has been minimized.

Copy link
Contributor Author

@rillig rillig commented Aug 2, 2019

@ianlancetaylor I took a snapshot of bootstrapping both go1.10.8 and go1.12.7, and the line with the highest CPU usage appeared when cmd/compile/internal/ssa was compiled:

VSZ: 947624
RSS: 700000
Time: 0:33.66

/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/pkg/tool/netbsd_amd64/compile \
-o \
/tmp/go-build924606097/b082/_pkg_.a \
-trimpath \
/tmp/go-build924606097/b082 \
-p \
cmd/compile/internal/ssa \
-std \
-complete \
-buildid \
U5uFnHi9i-u0_cTbqOyH/U5uFnHi9i-u0_cTbqOyH \
-D \
 \
-importcfg \
/tmp/go-build924606097/b082/importcfg \
-pack \
-c=3 \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/biasedsparsemap.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/block.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/branchelim.go \
... \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/writebarrier.go \
/home/rillig/tmp/pkgsrc.work/lang/go112/work/go/src/cmd/compile/internal/ssa/zcse.go

I'm impressed that it apparently takes 33 seconds and 700 MB to build this single package (if I'm interpreting the numbers correctly). In a previous build I didn't have swap memory enabled, and the build failed even earlier (the process 6g was killed then). Indeed, this is still less than 1 GB, so I must have measured something else when I initially wrote this issue.

Compared to go1.10.8, the largest process needed 27 MB more, which is a 4% increase. Since this is not much, it may have been pure coincidence that building go1.12 failed for me and building go1.10 had succeeded. Because of that difference I had opened this issue.

I still think 700 MB is quite a lot of memory for building a single go package, but that's probably already mentioned in several other issues, making this one redundant. Therefore I'm closing it.

@rillig rillig closed this Aug 2, 2019
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 2, 2019

Thanks for the info. CC'ing @randall77 as FYI, although the issue is now closed.

@randall77

This comment has been minimized.

Copy link
Contributor

@randall77 randall77 commented Aug 2, 2019

Yes, cmd/compile/internal/ssa is a very large package. It contains lots of autogenerated code, 375K lines in total. The memory used to compile it is something to keep an eye on, but I don't think we need to do anything at the moment. (Only Go developers need to compile this package, not Go users.) If we need to, it wouldn't be too hard to split out the autogenerated code into separate packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.