Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: Go 1.9 oom killed on compilation #21713

Closed
ybriffa opened this issue Aug 31, 2017 · 29 comments

Comments

Projects
None yet
9 participants
@ybriffa
Copy link

commented Aug 31, 2017

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user"
GORACE=""
GOROOT="/home/user/go"
GOTOOLDIR="/home/user/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build501174937=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

I compiled a very large code base (over 2000) of generated files, essentially composed of structure definition.

What did you expect to see?

Not to crash.

What did you see instead?

All my RAM filled, then my swap and finally the compilation got killed.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

I have 4Go RAM + 1Go swap

An information I forgot: I had no troubles under 1.8

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

it does not change anything. At the begining the compilation goes well, and at what looks like to be the end (maybe at the linking), the RAM and swap are filled and I get oom killed

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

Sure, here it is:
go build modelsources: /home/user/go/pkg/tool/linux_amd64/compile: signal: killed

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

read kernel buffer failed: Operation not permitted

I'm running under a dockerized environment

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

I can try, by now, what I can tell (with htop data) is that my RAM usage before the compilation is 648/3941MB and my swap is 440/1187MB

@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

[7263660.915777] xautolock invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[7263660.915781] xautolock cpuset=/ mems_allowed=0
[7263660.915783] CPU: 2 PID: 28341 Comm: xautolock Not tainted 3.13.0-83-generic #127-Ubuntu
[7263660.915784] Hardware name: Dell Inc. OptiPlex 7020/08WKV3, BIOS A02 11/20/2014
[7263660.915785]  0000000000000000 ffff880156213980 ffffffff81725992 ffff880210eb4800
[7263660.915787]  ffff880156213a08 ffffffff8171ff99 0000000000000000 0000000000000000
[7263660.915789]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[7263660.915791] Call Trace:
[7263660.915797]  [<ffffffff81725992>] dump_stack+0x45/0x56
[7263660.915799]  [<ffffffff8171ff99>] dump_header+0x7f/0x1f1
[7263660.915802]  [<ffffffff811531ee>] oom_kill_process+0x1ce/0x330
[7263660.915806]  [<ffffffff812d8995>] ? security_capable_noaudit+0x15/0x20
[7263660.915808]  [<ffffffff81153924>] out_of_memory+0x414/0x450
[7263660.915810]  [<ffffffff81159c20>] __alloc_pages_nodemask+0xa60/0xb80
[7263660.915813]  [<ffffffff811981e3>] alloc_pages_current+0xa3/0x160
[7263660.915816]  [<ffffffff8114fda7>] __page_cache_alloc+0x97/0xc0
[7263660.915817]  [<ffffffff811517b5>] filemap_fault+0x185/0x410
[7263660.915820]  [<ffffffff8117667f>] __do_fault+0x6f/0x530
[7263660.915822]  [<ffffffff8117a502>] handle_mm_fault+0x482/0xf10
[7263660.915827]  [<ffffffff8101260b>] ? __switch_to+0x16b/0x4d0
[7263660.915830]  [<ffffffff81731af4>] __do_page_fault+0x184/0x570
[7263660.915832]  [<ffffffff817291a2>] ? do_nanosleep+0x92/0x130
[7263660.915841]  [<ffffffff8108f9bd>] ? hrtimer_nanosleep+0xad/0x170
[7263660.915843]  [<ffffffff8108e840>] ? hrtimer_get_res+0x50/0x50
[7263660.915844]  [<ffffffff81731efa>] do_page_fault+0x1a/0x70
[7263660.915846]  [<ffffffff8172e228>] page_fault+0x28/0x30
[7263660.915847] Mem-Info:
[7263660.915848] Node 0 DMA per-cpu:
[7263660.915850] CPU    0: hi:    0, btch:   1 usd:   0
[7263660.915850] CPU    1: hi:    0, btch:   1 usd:   0
[7263660.915851] CPU    2: hi:    0, btch:   1 usd:   0
[7263660.915852] CPU    3: hi:    0, btch:   1 usd:   0
[7263660.915853] Node 0 DMA32 per-cpu:
[7263660.915854] CPU    0: hi:  186, btch:  31 usd:  58
[7263660.915854] CPU    1: hi:  186, btch:  31 usd: 152
[7263660.915855] CPU    2: hi:  186, btch:  31 usd: 172
[7263660.915856] CPU    3: hi:  186, btch:  31 usd: 162
[7263660.915856] Node 0 Normal per-cpu:
[7263660.915857] CPU    0: hi:  186, btch:  31 usd:  34
[7263660.915858] CPU    1: hi:  186, btch:  31 usd: 170
[7263660.915859] CPU    2: hi:  186, btch:  31 usd: 119
[7263660.915859] CPU    3: hi:  186, btch:  31 usd:  47
[7263660.915862] active_anon:1645345 inactive_anon:297726 isolated_anon:32
[7263660.915862]  active_file:121 inactive_file:0 isolated_file:0
[7263660.915862]  unevictable:8 dirty:0 writeback:23 unstable:0
[7263660.915862]  free:25234 slab_reclaimable:8094 slab_unreclaimable:12414
[7263660.915862]  mapped:110 shmem:62387 pagetables:10113 bounce:0
[7263660.915862]  free_cma:0
[7263660.915863] Node 0 DMA free:15852kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15852kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[7263660.915866] lowmem_reserve[]: 0 3414 7875 7875
[7263660.915868] Node 0 DMA32 free:46912kB min:29244kB low:36552kB high:43864kB active_anon:2826880kB inactive_anon:565304kB active_file:200kB inactive_file:92kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:3578448kB managed:3499420kB mlocked:0kB dirty:0kB writeback:20kB mapped:180kB shmem:102808kB slab_reclaimable:11292kB slab_unreclaimable:18000kB kernel_stack:1416kB pagetables:17176kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40625 all_unreclaimable? yes
[7263660.915871] lowmem_reserve[]: 0 0 4460 4460
[7263660.915872] Node 0 Normal free:38172kB min:38204kB low:47752kB high:57304kB active_anon:3754500kB inactive_anon:625600kB active_file:284kB inactive_file:0kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4700160kB managed:4567872kB mlocked:32kB dirty:0kB writeback:72kB mapped:260kB shmem:146740kB slab_reclaimable:21084kB slab_unreclaimable:31656kB kernel_stack:2528kB pagetables:23276kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:691 all_unreclaimable? yes
[7263660.915874] lowmem_reserve[]: 0 0 0 0
[7263660.915876] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15852kB
[7263660.915882] Node 0 DMA32: 1023*4kB (UEM) 1245*8kB (UE) 764*16kB (UEM) 395*32kB (UEM) 91*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 46788kB
[7263660.915887] Node 0 Normal: 888*4kB (EMR) 1559*8kB (UER) 762*16kB (UER) 195*32kB (UER) 14*64kB (UMR) 2*128kB (R) 2*256kB (R) 2*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 38168kB
[7263660.915894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[7263660.915894] 68172 total pagecache pages
[7263660.915895] 5633 pages in swap cache
[7263660.915896] Swap cache stats: add 9719686, delete 9714053, find 10458286/11131811
[7263660.915897] Free swap  = 0kB
[7263660.915898] Total swap = 997820kB
[7263660.915898] 2073648 pages RAM
[7263660.915899] 0 pages HighMem/MovableOnly
[7263660.915899] 33072 pages reserved
[7263660.915900] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[7263660.915907] [  495]     0   495    13026       27      29      227         -1000 systemd-udevd
[7263660.915912] [ 1549]   102  1549     9971      134      22      103             0 dbus-daemon
[7263660.915913] [ 1571]   101  1571    65728     2181      42     4806             0 rsyslogd
[7263660.915915] [ 1576]     0  1576    83621        0      66      815             0 ModemManager
[7263660.915916] [ 1594]     0  1594     4826        8      13       62             0 bluetoothd
[7263660.915917] [ 1601]     0  1601    17860      132      40     6960             0 systemd-logind
[7263660.915919] [ 1702]   104  1702     8088       27      20       49             0 avahi-daemon
[7263660.915920] [ 1707]   104  1707     8057        1      19       61             0 avahi-daemon
[7263660.915922] [ 1753]     0  1753    85327      150      62      325             0 NetworkManager
[7263660.915923] [ 1777]     0  1777    70269      650      41      428             0 polkitd
[7263660.915924] [ 1826]     0  1826     4021        1      13       37             0 getty
[7263660.915926] [ 1830]     0  1830     4021        1      14       38             0 getty
[7263660.915927] [ 1837]     0  1837     4021        1      13       38             0 getty
[7263660.915928] [ 1838]     0  1838     4021        1      12       38             0 getty
[7263660.915929] [ 1841]     0  1841     4021        1      13       38             0 getty
[7263660.915931] [ 1876]     0  1876    15347        1      32      169         -1000 sshd
[7263660.915932] [ 1897]     0  1897     5915       16      17       51             0 cron
[7263660.915933] [ 1914]     0  1914     4815       37      14       33             0 irqbalance
[7263660.915934] [ 1929]     0  1929    18875       13      41      221             0 cups-browsed
[7263660.915936] [ 1988]     0  1988     1094       12       8       41             0 acpid
[7263660.915937] [ 2034]   113  2034   107622        0      74      381             0 whoopsie
[7263660.915938] [ 2043]     0  2043     1884        3       8       27             0 gpm
[7263660.915940] [ 2059]   109  2059     9288       10      22       74             0 kerneloops
[7263660.915941] [ 2081]     0  2081    87908       62      39      167             0 lightdm
[7263660.915942] [ 2096]     0  2096    71865      628      43      147             0 accounts-daemon
[7263660.915944] [ 2197]     0  2197     6337       11      16       69             0 master
[7263660.915945] [ 2218]     0  2218    11236        1      25     1367             0 python2.7
[7263660.915946] [ 2242]     0  2242     3359       12      10       57             0 mdadm
[7263660.915947] [ 2260]     0  2260     3755       17      12       91             0 vmware-usbarbit
[7263660.915949] [ 2283]     0  2283    68378       27      34      505             0 vmware-view-usb
[7263660.915950] [ 2447]     0  2447    59837      605      48      182             0 upowerd
[7263660.915952] [ 2502]   111  2502    42230        0      20       59             0 rtkit-daemon
[7263660.915953] [ 2662]     0  2662    20072        1      45      141             0 login
[7263660.915954] [ 2672]     0  2672     2559       28       9      561             0 dhclient
[7263660.915955] [ 2948] 65534  2948     7823       16      21       49             0 dnsmasq
[7263660.915957] [ 3213]   108  3213    75370       20      47      866             0 colord
[7263660.915959] [19994]  4164 19994   104724      238      81      348             0 pulseaudio
[7263660.915960] [20260]     0 20260    93121      784      48      623             0 udisksd
[7263660.915962] [24648]   118 24648     6894       34      17       55             0 qmgr
[7263660.915963] [ 1440]  4164  1440    10219        0      23      593             0 zsh
[7263660.915964] [  409]     0   409     4872       19      14       34             0 upstart-udev-br
[7263660.915966] [  412]     0   412     3818       23      12       37             0 upstart-socket-
[7263660.915967] [  415]     0   415     3822       20      12       39             0 upstart-file-br
[7263660.915969] [11677]  4164 11677    11492        0      26     1857             0 zsh
[7263660.915970] [27964]  4164 27964    10171        0      24      516             0 zsh
[7263660.915971] [28011]     0 28011   112645     2182     119     3443             0 Xorg
[7263660.915972] [28098]     0 28098    42628        0      54      208             0 lightdm
[7263660.915974] [28112]  4164 28112    40225      597      82    13344             0 i3
[7263660.915975] [28255]  4164 28255     2659        2       8       81             0 ssh-agent
[7263660.915976] [28258]  4164 28258     6112        0      18       72             0 dbus-launch
[7263660.915978] [28259]  4164 28259     9887       78      23      108             0 dbus-daemon
[7263660.915979] [28330]  4164 28330     1114        1       7       25             0 sh
[7263660.915981] [28333]  4164 28333    92502      173      81      396             0 volumeicon
[7263660.915982] [28341]  4164 28341     5053       17      16       49             0 xautolock
[7263660.915983] [28457]  4164 28457    15878      100      38      133             0 xscreensaver
[7263660.915984] [28458]  4164 28458    15878      122      37      113             0 xscreensaver
[7263660.915986] [28569]  4164 28569     6528        0      19       76             0 zsh
[7263660.915987] [28575]  4164 28575   309084    36631     347    12265             0 pidgin
[7263660.915988] [28581]  4164 28581     9834       22      24       79             0 xfconfd
[7263660.915990] [28721]  4164 28721    48180       37      31      122             0 gvfsd
[7263660.915991] [28725]  4164 28725    86419        0      38      267             0 gvfsd-fuse
[7263660.915992] [28747]  4164 28747    13293       20      31       94             0 gconfd-2
[7263660.915993] [28749]  4164 28749    65915        0      30      173             0 at-spi-bus-laun
[7263660.915995] [ 8783]  4164  8783     1114        2       7       23             0 sh
[7263660.915996] [ 8784]  4164  8784    28711     3103      60     2803             0 urxvt
[7263660.915997] [ 8785]  4164  8785    20763        0      42      142             0 urxvt
[7263660.915999] [ 8786]  4164  8786    10352        1      23      706             0 zsh
[7263660.916000] [22183]  4164 22183    54795        5      37      182             0 gnome-keyring-d
[7263660.916001] [21699]  4164 21699    75201      668      52     1590             0 gvfs-udisks2-vo
[7263660.916002] [21704]  4164 21704    52129       46      38      151             0 gvfs-gphoto2-vo
[7263660.916004] [21708]  4164 21708    70512        1      39      175             0 gvfs-afc-volume
[7263660.916005] [21713]  4164 21713    49088       55      33       99             0 gvfs-mtp-volume
[7263660.916007] [ 9456]  4164  9456    88872       78      42      162             0 gvfsd-trash
[7263660.916008] [ 9471]  4164  9471    44547        0      23      657             0 dconf-service
[7263660.916009] [27457]  4164 27457     1114        1       7       24             0 sh
[7263660.916011] [27458]  4164 27458    28245       63      50     1224             0 urxvt
[7263660.916012] [27459]  4164 27459    20763        1      42      140             0 urxvt
[7263660.916013] [27460]  4164 27460    10244      184      24      412             0 zsh
[7263660.916014] [20253]  4164 20253     1114        1       7       24             0 sh
[7263660.916016] [20254]  4164 20254    29222      973      60     5455             0 urxvt
[7263660.916017] [20255]  4164 20255    20763        1      41      140             0 urxvt
[7263660.916018] [20256]  4164 20256    10292        0      26      654             0 zsh
[7263660.916020] [31407]  4164 31407     9785       25      25       66             0 dbus-daemon
[7263660.916021] [31410]  4164 31410    31231       33      29      121             0 at-spi2-registr
[7263660.916022] [ 4496]  4164  4496    30146        0      28      164             0 gvfsd-metadata
[7263660.916024] [30944]  4164 30944     1114        0       7       24             0 sh
[7263660.916025] [30945]  4164 30945    20997       91      44      158             0 i3bar
[7263660.916026] [30946]  4164 30946     1114        1       7       25             0 sh
[7263660.916027] [30947]  4164 30947     6547       27      18       43             0 i3status
[7263660.916029] [ 1631]  4164  1631     1114        1       7       25             0 sh
[7263660.916031] [ 1632]  4164  1632    28145      397      52     1406             0 urxvt
[7263660.916032] [ 1633]  4164  1633    20763        3      43      138             0 urxvt
[7263660.916033] [ 1634]  4164  1634    10417        0      24      755             0 zsh
[7263660.916034] [ 9730]  4164  9730     1114        1       7       24             0 sh
[7263660.916036] [ 9731]  4164  9731    28331      163      59     5343             0 urxvt
[7263660.916037] [ 9732]  4164  9732    20763        2      41      140             0 urxvt
[7263660.916038] [ 9734]  4164  9734    10307       92      25      573             0 zsh
[7263660.916039] [32088]     0 32088    19407       10      44      299             0 apache2
[7263660.916041] [ 4544]  4164  4544     1114        1       6       24             0 sh
[7263660.916042] [ 4545]  4164  4545    28143       60      50     1121             0 urxvt
[7263660.916043] [ 4546]  4164  4546    20763        0      41      141             0 urxvt
[7263660.916045] [ 4547]  4164  4547    10552      116      26      783             0 zsh
[7263660.916046] [18004]  4164 18004     1114        1       7       24             0 sh
[7263660.916047] [18005]  4164 18005    28685      929      54     2093             0 urxvt
[7263660.916048] [18006]  4164 18006    20763        1      41      140             0 urxvt
[7263660.916050] [18007]  4164 18007    10492       26      25      799             0 zsh
[7263660.916051] [27844]  4164 27844    10286        0      25      626             0 zsh
[7263660.916052] [  578]  4164   578     1114        1       7       25             0 sh
[7263660.916053] [  579]  4164   579    28647      559      55     1865             0 urxvt
[7263660.916055] [  580]  4164   580    20763        1      42      141             0 urxvt
[7263660.916056] [  581]  4164   581    10262        0      24      603             0 zsh
[7263660.916057] [  592]  4164   592    12070       18      29      168             0 ssh
[7263660.916058] [ 6750]  4164  6750     1114        1       7       25             0 sh
[7263660.916059] [ 6751]  4164  6751    28129       53      58     5259             0 urxvt
[7263660.916061] [ 6752]  4164  6752    20763        0      43      141             0 urxvt
[7263660.916062] [ 6753]  4164  6753    10411        9      25      742             0 zsh
[7263660.916064] [ 7139]  4164  7139     1114        1       7       24             0 sh
[7263660.916065] [ 7140]  4164  7140    28657       90      55     3011             0 urxvt
[7263660.916066] [ 7141]  4164  7141    20763        0      41      141             0 urxvt
[7263660.916067] [ 7142]  4164  7142    10310        1      24      651             0 zsh
[7263660.916068] [ 8586]  4164  8586    12070        6      25      185             0 ssh
[7263660.916070] [15488]  4164 15488     1114        1       8       24             0 sh
[7263660.916071] [15489]  4164 15489    26780      887      58     3078             0 urxvt
[7263660.916072] [15490]  4164 15490    20763        0      43      141             0 urxvt
[7263660.916074] [15491]  4164 15491     9676        0      25      532             0 zsh
[7263660.916075] [15499]  4164 15499    12109       21      28      172             0 ssh
[7263660.916076] [26984]  4164 26984     1114        2       7       23             0 sh
[7263660.916077] [26985]  4164 26985    28132       45      58     3381             0 urxvt
[7263660.916079] [26986]  4164 26986    20763        0      43      142             0 urxvt
[7263660.916080] [26987]  4164 26987    10262        0      24      608             0 zsh
[7263660.916081] [26994]  4164 26994    12206       31      28      292             0 ssh
[7263660.916082] [24362]  4164 24362     1114        1       6       25             0 sh
[7263660.916084] [24363]  4164 24363    28649      129      60     5715             0 urxvt
[7263660.916085] [24364]  4164 24364    20763        2      42      139             0 urxvt
[7263660.916086] [24365]  4164 24365    10319        0      26      679             0 zsh
[7263660.916088] [28395]  4164 28395     1114        1       7       24             0 sh
[7263660.916089] [28396]  4164 28396    28192      166      52     2334             0 urxvt
[7263660.916090] [28397]  4164 28397    20763        1      41      140             0 urxvt
[7263660.916092] [28398]  4164 28398    10528        1      26      866             0 zsh
[7263660.916093] [11339]  4164 11339     1114        1       7       24             0 sh
[7263660.916094] [11340]  4164 11340    28195      125      60     5248             0 urxvt
[7263660.916095] [11341]  4164 11341    20763        2      42      139             0 urxvt
[7263660.916096] [11342]  4164 11342    10449       68      24      726             0 zsh
[7263660.916098] [ 6344]  4164  6344    12350       21      28      443             0 ssh
[7263660.916099] [31001]     0 31001    19685       37      43      692             0 cupsd
[7263660.916100] [15080]  4164 15080    12109        8      28      205             0 ssh
[7263660.916102] [30856]  4164 30856    12110       15      26      178             0 ssh
[7263660.916103] [27963]  4164 27963    12070       17      28      169             0 ssh
[7263660.916104] [16035]  4164 16035    12150       85      25      182             0 ssh
[7263660.916105] [12868]    33 12868    91696        9      73      890             0 apache2
[7263660.916107] [12869]    33 12869    91696        9      73      380             0 apache2
[7263660.916108] [11045]  4164 11045     1114        2       7       23             0 sh
[7263660.916109] [11046]  4164 11046    28144      126      53     1161             0 urxvt
[7263660.916110] [11047]  4164 11047    20763        0      41      142             0 urxvt
[7263660.916112] [11048]  4164 11048    10294       69      24      575             0 zsh
[7263660.916113] [ 9830]  4164  9830    12070       41      26      145             0 ssh
[7263660.916114] [31792]  4164 31792    12070        3      27      184             0 ssh
[7263660.916116] [29439]  4164 29439     1114        2       7       23             0 sh
[7263660.916117] [29440]  4164 29440    28668     1889      59     3974             0 urxvt
[7263660.916118] [29441]  4164 29441    20763       16      41      126             0 urxvt
[7263660.916119] [29442]  4164 29442    10478        9      25      813             0 zsh
[7263660.916121] [ 1428]  4164  1428    12070        8      28      178             0 ssh
[7263660.916122] [12129]  4164 12129    12286      171      27      232             0 ssh
[7263660.916123] [31645]  4164 31645     1114       21       7        5             0 sh
[7263660.916124] [31646]  4164 31646    28169     2002      58     3345             0 urxvt
[7263660.916126] [31647]  4164 31647    20763      120      42       21             0 urxvt
[7263660.916127] [31648]  4164 31648    10307      504      24      160             0 zsh
[7263660.916128] [31669]  4164 31669     1114       25       6        1             0 sh
[7263660.916129] [31670]  4164 31670    28188     1006      52      222             0 urxvt
[7263660.916131] [31671]  4164 31671    20763      124      41       18             0 urxvt
[7263660.916132] [31672]  4164 31672    10307      177      26      487             0 zsh
[7263660.916133] [31736]  4164 31736    12109      172      28       42             0 ssh
[7263660.916134] [31737]  4164 31737    12070       56      26      130             0 ssh
[7263660.916136] [ 1108]  4164  1108     1114       20       7        5             0 sh
[7263660.916137] [ 1109]  4164  1109    28155     2088      59     3250             0 urxvt
[7263660.916139] [ 1110]  4164  1110    20763      132      42        9             0 urxvt
[7263660.916140] [ 1111]  4164  1111    10357      615      26       88             0 zsh
[7263660.916142] [14096]  4164 14096     1114       11       7       15             0 sh
[7263660.916143] [14097]  4164 14097    28182      908      54     2273             0 urxvt
[7263660.916144] [14098]  4164 14098    20763       51      42       91             0 urxvt
[7263660.916145] [14099]  4164 14099    10380      231      24      494             0 zsh
[7263660.916147] [30004]  4164 30004     1114       10       7       16             0 sh
[7263660.916148] [30005]  4164 30005    28204     2622      60     2707             0 urxvt
[7263660.916149] [30006]  4164 30006    20763       63      41       79             0 urxvt
[7263660.916151] [30007]  4164 30007    10285      157      25      482             0 zsh
[7263660.916152] [31317]   118 31317     6854        9      17       67             0 pickup
[7263660.916154] [32646]  4164 32646    63096       60      77      456             0 xfce4-notifyd
[7263660.916156] [ 3296]  4164  3296     6953      523      18       68             0 htop
[7263660.916158] [ 3917]  4164  3917    52850      855      25      807             0 go
[7263660.916159] [ 3946]  4164  3946  1923200  1804111    3760    92932             0 compile
[7263660.916160] Out of memory: Kill process 3946 (compile) score 837 or sacrifice child
[7263660.916162] Killed process 3946 (compile) total-vm:7692800kB, anon-rss:7216444kB, file-rss:0kB
@ybriffa

This comment has been minimized.

Copy link
Author

commented Aug 31, 2017

this one was under a 8Gb RAM + 1Gb Swap

@ianlancetaylor ianlancetaylor changed the title Go 1.9 oom killed on compilation cmd/compile: Go 1.9 oom killed on compilation Aug 31, 2017

@ianlancetaylor ianlancetaylor added this to the Go1.10 milestone Aug 31, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

@mdempsky

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

Did this code base compile with 1.8?

Also, it would help if we could reproduce this issue ourselves. Are you able to share the code base with us, or to create a new demo repo that exhibits the same issue?

@dcheney-atlassian

This comment has been minimized.

Copy link

commented Sep 1, 2017

Thanks for your patience; this could be due to two open issues, #21621, which is close to being closed, and #6643 which may be related, depending on what your generated code looks like.

@ybriffa

This comment has been minimized.

Copy link
Author

commented Sep 1, 2017

This code base used to compile with 1.8. I'm going to look if I can create you a repo to reproduce this problem with not-sensitive data.

I don' think it is related to the issue you've linked: I have exactly 2446 files. Also I tried to compile this code base over a continuous delivery software with a worker having 20Go RAM, it worked. However, I don't know exactly what is the threshold.

@ybriffa

This comment has been minimized.

Copy link
Author

commented Sep 1, 2017

After I have seen it with my company, I can provide you a tarball with the content of the code base a bit refined in order to debug. Since it is kind of sensitive content, where could I leave you the tarball ?

@mdempsky

This comment has been minimized.

Copy link
Member

commented Sep 19, 2017

@ybriffa Sorry for letting this slip. If it's not too large, you can just email it to me (mdempsky@google.com) as an attachment. Otherwise, upload to Google Drive or somewhere and share it with me.

@ybriffa

This comment has been minimized.

Copy link
Author

commented Oct 2, 2017

@mdempsky I've just sent you a tarball with the code we're using

@mdempsky

This comment has been minimized.

Copy link
Member

commented Oct 13, 2017

I received @ybriffa's test case. It's mostly self contained (only depends on github.com/mitchellh/mapstructure, which itself only depends on the standard library). It's 261015 lines of Go code spread over 2444 source files.

Bisecting on @ybriffa's test case finds e22ba7f to be the cause of the slow down and memory growth. Before that commit, building the package takes ~30 seconds and 2GB of memory on my workstation. After that commit, it takes ~70 seconds and >8GB.

According to -cpuprofile, it looks like most of the extra time is being spent in regalloc.

@mdempsky

This comment has been minimized.

Copy link
Member

commented Oct 13, 2017

I think the problem is there are two global map[string]func(...) (...) data structures that are initialized using the same 12,285 string literal keys.

@mdempsky

This comment has been minimized.

Copy link
Member

commented Oct 13, 2017

This Go program generates a Go source file that demonstrates the issue:

package main

import "fmt"

func main() {
        fmt.Printf("package p\n")
        for _, v := range []string{"a", "b"} {
                fmt.Printf("var %s = map[string]*int{\n", v)
                for i := 0; i < 12000; i++ {
                        fmt.Printf("\t\"%v\": nil,\n", i)
                }
                fmt.Printf("}\n")
        }
}
@mdempsky

This comment has been minimized.

Copy link
Member

commented Oct 13, 2017

@randall77

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2017

Seems to be fixed at tip. Almost certainly CL 66050, the fix for #22010 which initializes large maps like this (constant string keys, particularly) with tables instead of code.

Options:

  1. Backport CL 66050 to 1.9.
  2. Disable string CSE (CL 28219) for 1.9.
  3. Do nothing.

I'm kinda leaning toward #3. These large files are certainly generated code (well, @mdempsky 's repro is, not 100% sure about the original bug). There is a workaround, just do what 1 does by hand, initialize an array instead and load the data into a map with a for loop.

@mdempsky

This comment has been minimized.

Copy link
Member

commented Oct 16, 2017

@randall77 Changing *int and nil to func() and func() {} makes the test case more representative of the original repro, and still takes a lot more memory than Go 1.8 did.

We should be able to table-initialize that too, since global funcs don't need closures. But it seems like we don't for some reason.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2017

@mdempsky I'll take a look at that.

In any case, the question of what to do about 1.9 still stands.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 16, 2017

Change https://golang.org/cl/71210 mentions this issue: cmd/compile: more static function literal optimizations

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2017

@randall77, will let you decide but everyone at proposal-review is OK with option 3 "do nothing".

@randall77

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2017

I'll call this done then.

@randall77 randall77 closed this Dec 4, 2017

@golang golang locked and limited conversation to collaborators Dec 4, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.