Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: reported memory usage discrepancy (61%) between Go runtime and OS #30904

Closed
karalabe opened this issue Mar 18, 2019 · 4 comments

Comments

Projects
None yet
3 participants
@karalabe
Copy link
Contributor

commented Mar 18, 2019

What version of Go are you using (go version)?

$ go version
1.12.1

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Alpine docker container on top of Ubuntu.

What did you do?

After updating to Go 1.12, our memory usage pattern changed quite a lot. The amount of memory held by Go for the heap seemingly went down a bit and we can see it being relinquished more aggressively, if we are plotting runtime.MemStats.HeapSys - runtime.MemStats.HeapReleased .

Screenshot from 2019-03-18 16-08-01

The light purple and light green lines are the heap usage as reported by runtime.MemStats.HeapUsed (light purple = Go 1.12.1, light green = Go 1.11.5). The purple one uses 512MB more memory (not exactly the same code underneath), but that's not so relevant here.

The dark purple (Go 1.12.1) and dark blue (Go 1.11.5) lines are the heap total as reported by HeapSys - HeapReleased. This cleanly shows that Go 1.12 is more aggressive, as the "wasted heap" is a lot less compared to the used heap + relinquishing is also jaggedy compared to Go 1.11.5's flat curve.

If however we plot the amount of memory held by Go as seen by the OS (e.g. top, free or other process utilities), we get a 61% increase compared to Go 1.11.5. The top chart below (Memory (Master)) is Go 1.11.5's memory usage as reported by the OS for the same interval as the above heap charts. The bottom chart (Memory (EXP)) is Go 1.12.1's memory usage.

Screenshot from 2019-03-18 16-08-05
Screenshot from 2019-03-18 16-08-05b

You can clearly see that Go 1.11.5's heap report matches with the OS's memory usage report. However, Go 1.12.1 reports vastly less memory being used than the OS reports. Based on our own production code, I tend to believe that the Go 1.12.1 heap charts are correct (i.e. they are in line with the 1.11.5 ones), however, this means Go is keeping hold of something that it itself is not aware of.

@karalabe karalabe changed the title Reported memory usage discrepancy between Go runtime and OS Reported memory usage discrepancy (61%) between Go runtime and OS Mar 18, 2019

@ALTree

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

1.12 release notes:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed. To revert to the Go 1.11 behavior (MADV_DONTNEED), set the environment variable GODEBUG=madvdontneed=1.

Does the old behaviour return if you set GODEBUG=madvdontneed=1?

@karalabe

This comment has been minimized.

Copy link
Contributor Author

commented Mar 18, 2019

Ah, didn't know about this. That might just explain it. Is there a way to ask the OS how much memory is in such a limbo state?

@ALTree

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

Is there a way to ask the OS how much memory is in such a limbo state?

I personally don't know one, but this is more a Linux question than a Go one.

The runtime change is well documented in the release notes and the GODEBUG flag lets you override the new behaviour; so I don't think there's anything more to do here. I suggest we close this issue, unless you think this should be better documented and want to turn this into a doc bug.

@ALTree ALTree changed the title Reported memory usage discrepancy (61%) between Go runtime and OS runtime: reported memory usage discrepancy (61%) between Go runtime and OS Mar 18, 2019

@agnivade

This comment has been minimized.

Copy link
Member

commented Mar 19, 2019

Seems like this is resolved. Please feel free to reopen if you think we need to document this somewhere else too.

@agnivade agnivade closed this Mar 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.