Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: 1.3MB program takes 290MB of memory to compile #1860

Open
davecheney opened this issue May 21, 2011 · 12 comments

Comments

@davecheney
Copy link
Contributor

commented May 21, 2011

When compiling pkg/exp/eval or test/64bit.go, RSS memory usage using 5g or 6g exceeds
100mb. 

What is the expected output?

RSS usage should be more inline with other packages, around 10-20 mb

What do you see instead?

When compiling pkg/exp/eval or test/64bit.go, RSS memory usage using 5g or 6g exceeds
100mb. 

Which compiler are you using (5g, 6g, 8g, gccgo)?

5g and 6g

Which operating system are you using?

Debian 6 amd64 and arm5

Which revision are you using?  (hg identify)

+tip

Please provide any additional information below.
@rsc

This comment has been minimized.

Copy link
Contributor

commented May 23, 2011

Comment 1:

It is probably not test/64bit.go but its *output* that 6g uses a lot
of memory to compile.
That's a big program.

Status changed to LongTerm.

@davecheney

This comment has been minimized.

Copy link
Contributor Author

commented May 24, 2011

Comment 2:

Ahh yes, tmp.go is 1,300kb of source, which consumes 289,145,039 bytes (according to
valgrind). Looking at the way subr.c:mal works this isn't an easy problem to fix (or a a
very high priority one)
@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2011

Comment 3:

Labels changed: added priority-later.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2012

Comment 4:

I would like to make constants use int64 until it is necessary to use a bignum.
That would probably fix this issue.

Labels changed: added performance.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2012

Comment 5:

Labels changed: added go1.1maybe.

@remyoudompheng

This comment has been minimized.

Copy link
Contributor

commented Dec 23, 2012

Comment 6:

I am starting having problems to compile go on a amd64 machine I don't use really often.
$ uname -a
Linux maison 3.7.1-2-ARCH #1 SMP PREEMPT Thu Dec 20 17:57:00 CET 2012 x86_64 GNU/Linux
$ sysctl vm.overcommit_memory
vm.overcommit_memory = 1
$ free -h
             total       used       free     shared    buffers     cached
Mem:          998M       675M       322M         0B        13M        88M
-/+ buffers/cache:       573M       424M
Swap:           0B         0B         0B
$ cat /proc/cpuinfo
[...]
model name  : Intel(R) Pentium(R) D CPU 2.80GHz
[...]
cpu MHz     : 2792.945
[...]
cpu cores   : 2
[...]
$ go install -a -v std
[...]
compress/lzw
hash/adler32
compress/zlib
container/list
container/ring
crypto/sha512
database/sql/driver
database/sql
debug/gosym
encoding/ascii85
cmd/go
expvar
net/http/pprof
cmd/godoc
encoding/base32
go build encoding/base32: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot
allocate memory
encoding/csv
go build encoding/csv: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/cookiejar
go build exp/cookiejar: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/ebnf
go build exp/ebnf: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate memory
exp/types
go build exp/types: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/html/atom
go build exp/html/atom: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
[...]
Restoring vm.overcommit_memory=1 solves the problem. The case described by this issue
happens for real in the standard library now:
$ ls -l src/pkg/exp/locale/collate
total 2892
drwxr-xr-x 2 remy remy    4096 Dec 23 21:28 build
-rw-r--r-- 1 remy remy    7284 Dec 23 21:28 colelem.go
-rw-r--r-- 1 remy remy    4368 Dec 23 21:28 colelem_test.go
-rw-r--r-- 1 remy remy   12571 Dec 23 21:28 collate.go
-rw-r--r-- 1 remy remy   13089 Dec 23 21:28 collate_test.go
-rw-r--r-- 1 remy remy    2392 Dec 23 21:28 contract.go
-rw-r--r-- 1 remy remy    2291 Dec 23 21:28 contract_test.go
-rw-r--r-- 1 remy remy    1112 Dec 23 21:28 export.go
-rw-r--r-- 1 remy remy    1957 Dec 23 21:28 export_test.go
-rw-r--r-- 1 remy remy     403 Dec 23 21:28 Makefile
-rw-r--r-- 1 remy remy   17770 Dec 23 21:28 maketables.go
-rw-r--r-- 1 remy remy    7009 Dec 23 21:28 regtest.go
-rw-r--r-- 1 remy remy    3914 Dec 23 21:28 table.go
-rw-r--r-- 1 remy remy 2828722 Dec 23 21:28 tables.go
-rw-r--r-- 1 remy remy    8641 Dec 23 21:28 table_test.go
drwxr-xr-x 3 remy remy    4096 Dec 23 21:28 tools
-rw-r--r-- 1 remy remy    2264 Dec 23 21:28 trie.go
-rw-r--r-- 1 remy remy    2596 Dec 23 21:28 trie_test.go
@robpike

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2013

Comment 8:

Labels changed: removed go1.1maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2013

Comment 9:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2013

Comment 10:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2013

Comment 11:

Labels changed: added repo-main.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc changed the title cmd/gc: 1.3MB program takes 290MB of memory to compile cmd/compile: 1.3MB program takes 290MB of memory to compile Jun 8, 2015

@bankole7782

This comment has been minimized.

Copy link

commented Apr 24, 2018

This doesn't seem to be a problem. Computers are not in the days of 512mb RAM but 8gb RAM common place.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2018

A large multiplier in resource use when compiling is always a problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.