runtime: high startup memory usage of 64-bit executable on Windows

[gopherbot]

by sjbogdan:

A 64-bit executable consumes 540 MB on "hello world" project on startup
32-bit executable (on x86-64 system) uses acceptable amount of memory ~ 9 MB

Version affected : v1.1beta2 and v1.1rc1 compiles

> go version
go version devel +bea6199b09ea Tue Apr 30 17:47:39 2013 -0700 windows/amd64

Code : any hello world program compiled with 6g
Example :
package main

import "time"

func main () {
    time.Sleep ( 1 * time.Minute )
}

---
See attachments

Additional information :
Windows 8 ( x86-64 ) Enterprize
Never used intaller, extracted from zip and set up GOROOT & GOPATH

Attachments:

1. ResMonitor.png (18103 bytes)
2. PM_amd64.png (19327 bytes)
3. PM1_amd64.png (9353 bytes)
4. PM_386.png (19339 bytes)
5. PM1_386.png (9393 bytes)

[remyoudompheng]

Comment 1:

It looks like virtual memory. Try running 50 copies of the program and see if it uses
25GB of RAM or <1GB.
See also issue #5236.

[minux]

Comment 2:

does this happen on other windows version?
I tested on windows server 2008 R2, but failed to reproduce.

[gopherbot]

Comment 3 by sjbogdan:

8 running processes ate 5 GB of swap / commit ( see attachment )
Can't test on other systems, attaching executable ( note : 64-bit executable )

Attachments:

1. PM_amd64_8proc.png (57716 bytes)
2. 1.exe (548352 bytes)

[remyoudompheng]

Comment 4:

The amount of used physical memory does not seem to have changed in your graph. Can you
run 50 processes, as asked before, without problems?

[gopherbot]

Comment 5 by sjbogdan:

NO, program won't start without swap available
( program window doesn't show up, then Windows suggests to close other programs )

Attachments:

1. Untitled.png (9736 bytes)

[minux]

Comment 6:

is your windows system set to disable swap (page file)?

[minux]

Comment 7:

note:
64-bit Go program do allocate about 0.5 GB memory for it's heap metadata on
startup. But as the helloworld program shouldn't touch most of them, they
won't be backed by physical memory normally.
however, things might change if you disable swap (page file) completely.

[gopherbot]

Comment 8 by sjbogdan:

> is your windows system set to disable swap (page file)?
No
Can you run & show "Process Explorer"s memory info of my executable ?

[gopherbot]

Comment 9 by sjbogdan:

> 64-bit Go program do allocate about 0.5 GB memory for it's heap metadata on startup. >
But as the helloworld program shouldn't touch most of them, they won't be backed by
physical memory normally.
where can I read about it ?
it there any way to influence this ? ( build flags / system variables / other settings ?
)

[remyoudompheng]

Comment 10:

I linked to issue #5236. Did you read it at all?

[minux]

Comment 11:

the relevant code is here:
https://golang.org/cl/7307122/diff/12002/src/pkg/runtime/malloc.goc#newcode316
my memory was wrong, the figure should be about 256 MB on amd64, and i just verified
that the mheap structure is 268480448 bytes on my system.
sorry, i don't have a 64-bit windows VM available to run arbitrary binaries. could you
please
verify that the same applies to godoc.exe in go1.1rc1 binary distribution?
you could run "godoc.exe -http localhost:6060" and then view its memory usage.

[gopherbot]

Comment 12 by sjbogdan:

> I linked to issue #5236. Did you read it at all?
Yes
- swap is not disabled ( a lot of talks and theory, not a single phrase about high
virtual memory usage )
- there is no patch for Windows amd64
- I'm not a linux system programmer, so can't tell whether it relates to Windows directly
- why there is such a huge difference between 8g vs 6g on Windows ?

[gopherbot]

Comment 13 by sjbogdan:

godoc

Attachments:

1. PM_amd64_godoc.png (19871 bytes)
2. PM1_amd64_godoc.png (9983 bytes)

[minux]

Comment 14:

ok, i reproduced this using sysinternal's process explorer.
this is working as intended at least for now.
go 1.0.3 use 70MB committed memory at startup, go 1.1rc1 will use ~500MB.
go 1.0.3 could use at most 16GB heap whereas go 1.1rc1 supports 128GB,
so the committed memory usage is proportionate to maximum heap size.
this has something to do with windows memory management (it will reserve
page file even for memory that don't actually used.)
yes, this does mean that you can't run a large number of helloworld Go
programs on 64-bit windows without allocating a very large page file.
i will accept this issue, but i doubt we have simple solutions to this.
(we can lazily reserve the heap metadata, but that surely won't happen
before Go 1.1, this is issue #5236)
We need to investigate why the binary uses ~500MB of committed memory
instead of ~256MB as needed on unix.
after that, we can merge this issue with issue #5236.

Labels changed: added priority-later, expertneeded, removed priority-triage.

Status changed to Accepted.

[minux]

Comment 16:

the issue tracker ate my reply two times.
i instrumented all calls to VirtualAlloc in pkg/runtime/mem_windows.c, and running
helloworld
program produces this trace:
<alloc 131072 bytes>
<alloc 268480448 bytes>
<reserve 146028888064 bytes>
<alloc 131072 bytes>
<map 1048576 bytes>
<map 65536 bytes>
<alloc 131072 bytes>
<alloc 65536 bytes>
<alloc 131072 bytes>
<alloc 4080 bytes>
<alloc 1048576 bytes>
This still cannot explain the 540MB of committed memory usage.