runtime/debug: FreeOSMemory does not return memory on 64-bit Windows with 386 release

[gopherbot]

by mgspross:

When running go1.3.3 windows/386 on a 64-bit Windows 7 machine, FreeOSMemory does not
return memory back to the operating system.

FreeOSMemory works correctly with go1.3.3 windows/amd64 on the same machine.

What does 'go version' print?
go version go1.3.3 windows/386

What steps reproduce the problem?
Run the attached go program (memtest.go) on a 64-bit Windows machine, using go1.3.3
windows/386:

    set GCDEBUG=gctrace=1 && go run memtest.go

After 5 seconds, the program allocates ~95 MB of memory via a slice. The program then
waits for user input and attempts to release memory with FreeOSMemory periodically.

What happened?
Memory is not returned to the operating system. Running the test program with
GODEBUG=gctrace=1 shows the following, with no change after multiple calls to
FreeOSMemory over a period of time:

    inuse: 95, idle:0, sys: 96, released: 0, consumed: 95 (MB)

What should have happened instead?
FreeOSMemory should return memory back to the OS.

Additional information

The amount of memory actually returned to the OS varies with different go releases as
well (I tested the most recent releases). Here are my findings, running memtest.go with
"go run memtest.go" on various go releases:

    * go1.3.2 windows/386: The memory is never returned to the OS.
    * go1.3.2 windows/amd64: This works correctly. The memory is released and the memory usage returns to close to what it was when the program started (memory in-use was ~1.5MB after FreeOSMemory was called).
    * go1.3.3 windows/386: The memory is never returned to the OS.
    * go1.3.3 windows/amd64: This works correctly (memory in-use was ~1.5MB after FreeOSMemory was called).
    * go1.4rc1 windows/386: Some memory is returned, bu not all of it: ~13.5MB was still in use after FreeOSMemory was called.
    * go1.4rc1 windows/amd64: Some memory is returned, but not all of it: ~7.6MB was still in use after FreeOSMemory was called.

These results were all repeatable which each respective release.

P.S. I'm using the 386 build because my the program I'm writing uses interacts with
legacy 32-bit Windows COM libraries, so I can't use the amd64 build in my current
situation.

Attachments:

1. memtest.go (647 bytes)

[ianlancetaylor]

Comment 1:

Labels changed: added repo-main, release-go1.5, os-windows.

[gopherbot]

Comment 2 by mgspross:

A follow-up: I just tested this on a 32-bit Windows 7 installation, using go1.3.3
windows/386, and got the same behavior (memory is not returned to the OS), so the issue
occurs on both 32-bit and 64-bit versions of Windows with the 386 build, in case that
helps to narrow down the root cause.

[alexbrainman]

Comment 3:

mgspross,
I cannot reproduce you problem. I have go version close to go1.4
changeset:   21866:ffe33f1f1f17
user:        Andrew Gerrand <adg@golang.org>
date:        Tue Nov 25 15:41:33 2014 +1100
summary:     doc: tidy up "Projects" page; add Go 1.4
If I build your program for windows/386. And then run it on 64-bit Windows 7 like this:
set GODEBUG=gctrace=1 && c:\test
I get this output:
gc1(1): 0+0+5859+0 us, 0 -> 0 MB, 65 (65-0) objects, 2 goroutines, 15/0/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
gc2(1): 0+0+0+0 us, 0 -> 0 MB, 295 (300-5) objects, 4 goroutines, 21/0/0 sweeps, 0(0)
handoff, 0(0) steal, 0/0/0 yields
5 second delay
allocating large slice
gc3(1): 0+0+976+0 us, 0 -> 95 MB, 330 (369-39) objects, 6 goroutines, 23/0/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
Large slice set to nil
Press any key to terminate...
Freeing memory...
gc4(1): 0+0+976+0 us, 95 -> 95 MB, 317 (387-70) objects, 6 goroutines, 23/2/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: 96 MB released
scvg-1: inuse: 0, idle: 96, sys: 96, released: 96, consumed: 0 (MB)
Freeing memory...
gc5(1): 0+0+0+0 us, 81 -> 0 MB, 313 (395-82) objects, 6 goroutines, 23/0/21 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: inuse: 0, idle: 96, sys: 96, released: 96, consumed: 0 (MB)
Freeing memory...
gc6(1): 0+0+0+0 us, 0 -> 0 MB, 313 (403-90) objects, 6 goroutines, 23/0/20 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: inuse: 0, idle: 96, sys: 96, released: 96, consumed: 0 (MB)
Freeing memory...
gc7(1): 0+0+976+0 us, 0 -> 0 MB, 313 (411-98) objects, 6 goroutines, 23/0/20 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: inuse: 0, idle: 96, sys: 96, released: 96, consumed: 0 (MB)
And that is different from yours. Perhaps it has been fixed?
Alex

Status changed to WaitingForReply.

[gopherbot]

Comment 4 by mgspross:

Alex, 
Yes, it's possible this was fixed already, but I was a little confused by the different
results I was getting for different releases. When I tested go1.4rc1 windows/386, it
still didn't seem to release all the memory. I'll try again when I'm near my laptop.
Is your 1.4 newer or older than 1.4rc1? Maybe I need to test with go tip?

[alexbrainman]

Comment 5:

My version is current tip. But, I am pretty sure, 1.4rc1 is good enough.
Alex

[gopherbot]

Comment 6 by mgspross:

OK. I tested again with go1.4rc1 windows/386, and I got the same output from
GODEBUG=gctrace=1 that you posted. Specifically, I saw this line, which is what I would
expect to see:
   scvg-1: 96 MB released
I apologize: when I initially wrote the bug report, I had been paying more attention to
the process's memory usage in the Windows task manager, and I missed the above line in
the gctrace output.
When I run the test with 1.3.3 windows/386, the output reports that 0 MB are released,
so perhaps this different in 1.4 because of the changes to the garbage collector.
So, this does seem to be working in 1.4, as far as what the go runtime is reporting.
However, in 1.4, what still seems odd is that Windows reports that ~13.5 is still "in
use" after FreeOSMemory is called (in the "Private Working Set" column of the Task
Manager). When the program starts, it only uses a little over 1 MB.
My concern is whether this has any implications for long-running go programs (i.e. a
server). I wonder if a go program can still run out of memory eventually on Windows,
since Windows doesn't seem to decommit all of the memory, even though the runtime says
it has released all of it.
What makes it seem strange is that this doesn't happen with the amd64: there is no
"leftover" memory reported by the Task Manager, or at least not a significant amount:
the memory usage goes back down to ~1.5 MB, according to the Task Manager.
So, I'm not sure if this is somehow normal behavior in the 386 build, or if this means
the 386 build is leaking memory in 1.4.

[alexbrainman]

Comment 7:

> ... So, this does seem to be working in 1.4, as far as what the go runtime is
reporting.
Closing this issue then.
> ... However, in 1.4, what still seems odd is that Windows reports that ~13.5 is still
"in use" ...
The way runtime works is not straight froward (and changing all the time) - there are
many factors involved. Runtime creates its own structures. You can investigate
everything yourself. windows memory interface lives in
%GOROOT%\src\runtime\mem_windows.go. You could put printf statements there to see what
happens in your program. If you get stuck or need some explaining - everyone here (or on
go-nuts) will help.
You should use vmmap program to measure Windows memory usage. Task Manager provides very
little details.
Alex

Status changed to Retracted.