Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: GC: heap idle is not released / decrease #14521

Closed
tonglil opened this issue Feb 26, 2016 · 9 comments

Comments

@tonglil
Copy link

commented Feb 26, 2016

This is my first time submitting to Go, apologies in advance if it is a non-helpful issue.

  1. What version of Go are you using (go version)?
    1.6
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOOS="darwin"
    CC="clang"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
    CXX="clang++"
  3. What did you do?
    (Use play.golang.org to provide a runnable example, if possible.)

Sorry in advance, I am not able to extract a runnable example at the moment as it is a larger program I am debugging.

I am having issues with the GC in Go. Specifically, it does not seem to release idle heap memory back to the system, even after +10 minutes of inactivity (based on this) in the code/area that generated the initial garabage in the program.

Heap in use and allocated match one another (makes sense).
Heap sys = heap idle (this does not make sense to me).

  1. What did you expect to see?
    Heap idle to decrease.
  2. What did you see instead?
    Heap idle does not decrease over time. I even called runtime.GC() and then debug.FreeOSMemory() as well in attempt to release the memory.

Other information
Memory profile information from MemStats logged every 2 seconds:
picture1

I did reading of different sources and it one thought I see is that the GC may release memory, but it is up to the OS to collect/ask for it. Is this correct? Does this mean the memory footprint will not decrease without external intervention?

I thought maybe this problem has to do with #13552, however from 1.5 to 1.6 made no difference for me.

Thank you for your time.

@rubycut

This comment has been minimized.

Copy link

commented Feb 26, 2016

I also called FreeOsMemory(), nothing happends on linux, golang 1.5.3

10:17:52 WARN Allocated memory before GC: 3136 Mb
10:17:52 WARN Allocated memory from OS before FreeOSMemory: 3323 Mb
10:18:02 WARN Allocated memory after GC: 1599 Mb
10:18:02 WARN HeapInusde memory: 1712 Mb
10:18:02 WARN HeapIdle memory: 1471 Mb
10:18:02 WARN HeapReleased memory: 1471 Mb
10:18:02 WARN HeapSys memory: 3184 Mb
10:18:02 WARN Allocated memory from OS: 3323 Mb

@kaypour

This comment has been minimized.

Copy link

commented Feb 26, 2016

Hmm, I am not sure but afaik runtime.GC() and debug.FreeOSMemory() are not guaranteed to release memory, since Go's GC is conservative, so it keeps memory, even if it is not in use, incase it has to allocate again. (Someone else correct me, if I am wrong. Just my observations)
Does your application crash with out of memory?

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2016

The go runtime can only advise the operating system that a section of
virtual address space is no longer needed, it cannot force the operating
system to take action. Operating systems, especially linux, and especially
if they are not under significant memory pressure can, and regularly do,
ignore this advice.

The best way to determine the memory usage of your application is via
net/http/pprof, or by enabling GODEBUG=gctrace=1

On Fri, Feb 26, 2016 at 10:43 PM, Daniel Kaypour notifications@github.com
wrote:

Hmm, I am not sure but afaik runtime.GC() and debug.FreeOSMemory() are not
guaranteed to release memory, since Go's GC is conservative, so it keeps
memory, even if it is not in use, incase it has to allocate again. (Someone
else correct me, if I am wrong. Just my observations)
Does your application crash with out of memory?


Reply to this email directly or view it on GitHub
#14521 (comment).

@bradfitz

This comment has been minimized.

Copy link
Member

commented Feb 26, 2016

@Opposition, Go's GC is not conservative. It's a precise GC.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2016

HeapReleased is how much memory Go did give back to the OS. Go has no way of knowing if the OS did actually take the memory back, so it has no stat about that.

HeapIdle includes the HeapReleased stat. So HeapIdle should not go down when Go gives the memory back to the OS.

@rubycut

This comment has been minimized.

Copy link

commented Feb 26, 2016

Problem is if linux will not take memory back, after some time process get
oom killed.

On Fri, Feb 26, 2016 at 6:04 PM Keith Randall notifications@github.com
wrote:

HeapReleased is how much memory Go did give back to the OS. Go has no way
of knowing if the OS did actually take the memory back, so it has no stat
about that.

HeapIdle includes the HeapReleased stat. So HeapIdle should not go down
when Go gives the memory back to the OS.


Reply to this email directly or view it on GitHub
#14521 (comment).

@ianlancetaylor ianlancetaylor changed the title GC: heap idle is not released / decrease runtime: GC: heap idle is not released / decrease Feb 26, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2016

I'm not sure we can make progress on this without an example that we can run ourselves.

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Feb 26, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2016

If possible, you may want to try running your program under truss (or whatever it is called on Darwin) to confirm that the Go program is calling madvise as expected. That is the signal the Go runtime uses to tell the OS that it no longer needs some memory pages.

@tonglil

This comment has been minimized.

Copy link
Author

commented Feb 26, 2016

Thank you all for your help.

I will run this on Linux and use truss or otherwise to see if I can get the OS to reclaim the memory and reduce the heap idle amount that way.

Please feel free to close issue, I can update and reopen it at a later time with something more concrete for you to go on :)

On Feb 26, 2016, at 9:28 AM, Ian Lance Taylor notifications@github.com wrote:

If possible, you may want to try running your program under truss (or whatever it is called on Darwin) to confirm that the Go program is calling madvise as expected. That is the signal the Go runtime uses to tell the OS that it no longer needs some memory pages.


Reply to this email directly or view it on GitHub.

@bradfitz bradfitz closed this Feb 26, 2016

@tidwall tidwall referenced this issue Dec 22, 2016

@golang golang locked and limited conversation to collaborators Feb 28, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
8 participants
You can’t perform that action at this time.