Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: GC: heap idle is not released to linux #33376

Open
Anteoy opened this issue Jul 31, 2019 · 4 comments

Comments

@Anteoy
Copy link

commented Jul 31, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

No verification

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
local env for build:
GOARCH="amd64"
GOBIN="/home/zhoudazhuang/gobin/"
GOCACHE="/home/zhoudazhuang/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/zhoudazhuang/class100/gtools:/home/zhoudazhuang/goproject"
GOPROXY=""
GORACE=""
GOROOT="/home/zhoudazhuang/usr/local/go1.12.1/go"
GOTMPDIR=""
GOTOOLDIR="/home/zhoudazhuang/usr/local/go1.12.1/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build499587841=/tmp/go-build -gno-record-gcc-switches"

online server:
╰─># uname -a
Linux ll-025048236-FWWG.AppPZFW.prod.bj1 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
╰─># cat /etc/issue
CentOS release 6.8 (Final)
Kernel \r on an \m

What did you do?

Normal operation, heapIdle is getting bigger and bigger

gc 24015 @413998.746s 23%: 16+3647+0.43 ms clock, 325+52790/18232/0+8.6 ms cpu, 6873->6900->5671 MB, 11404 MB goal, 20 P (forced)
scvg-1: 151 MB released
scvg-1: inuse: 9745, idle: 52574, sys: 62319, released: 52574, consumed: 9745 (MB)
gc end: heapSys->65346568192, heapAlloc->6780874680, heapIdle->55123410944, heapReleased->55123247104
gc 24016 @414005.492s 23%: 7.1+3727+0.16 ms clock, 142+52631/18634/0+3.2 ms cpu, 6722->6838->5787 MB, 11342 MB goal, 20 P (forced)
scvg-1: 149 MB released
scvg-1: inuse: 9730, idle: 52594, sys: 62325, released: 52594, consumed: 9730 (MB)
gc end: heapSys->65356005376, heapAlloc->6708748256, heapIdle->55151747072, heapReleased->55149142016
scvg2759: inuse: 9818, idle: 52491, sys: 62309, released: 52491, consumed: 9818 (MB)
gc 24017 @414011.980s 23%: 21+3679+0.22 ms clock, 438+52629/18393/0+4.4 ms cpu, 6651->6707->5694 MB, 11575 MB goal, 20 P (forced)
scvg-1: 142 MB released
scvg-1: inuse: 9685, idle: 52629, sys: 62315, released: 52629, consumed: 9685 (MB)
gc end: heapSys->65344962560, heapAlloc->6548453928, heapIdle->55188455424, heapReleased->55185702912
gc 24018 @414018.410s 23%: 25+3701+0.37 ms clock, 506+52583/18491/0+7.4 ms cpu, 6494->6558->5660 MB, 11389 MB goal, 20 P (forced)
scvg-1: 177 MB released
scvg-1: inuse: 9599, idle: 52687, sys: 62286, released: 52687, consumed: 9599 (MB)
gc end: heapSys->65312587776, heapAlloc->6380088128, heapIdle->55247020032, heapReleased->55246462976

And the MemStats:

fmtdebug Mem stats: {Alloc:7923150672 TotalAlloc:87802159107216 Sys:83425758664 Lookups:0 Mallocs:1317080204050 Frees:1316977196669 HeapAlloc:7923150672 HeapSys:63562973184 HeapIdle:52234674176 HeapInuse:11328299008 HeapReleased:51822125056 HeapObjects:103007381 StackInuse:16269934592 StackSys:16269934592 MSpanInuse:336623760 MSpanSys:420052992 MCacheInuse:34720 MCacheSys:49152 BuckHashSys:2346567 GCSys:2769477632 OtherSys:400924545 NextGC:12386391312 LastGC:1564479987316535864 PauseTotalNs:368820215369 PauseNs:[1287681 4999082 4236046 35185381 28689620 21532347 4224238 11823736 27419782 32950552 25486288 19654996 31435873 4595087 20290070 1704411 14529638 11940630 6121778 25226722 3802688 1190

What did you expect to see?

heapIdle should release to os

What did you see instead?

heapIdle is not released to os , even thougth I periodic call debug.FreeOSMemory()

@Anteoy

This comment has been minimized.

Copy link
Author

commented Jul 31, 2019

#14521 Maybe this is relevant. But my linux kernel is 2.6. I am not sure if the madvise system call will use madvdontneed, I seted the evn 'madvdontneed=1'

@Anteoy

This comment has been minimized.

Copy link
Author

commented Jul 31, 2019

Another thing:

proc.go sysmon() 
 // If a heap span goes unused for 5 minutes after a garbage collection,
// we hand it back to the operating system.

MemStats reported the heapIdle is 52234674176, And it will not decrease when I periodic call debug.FreeOSMemory(). But how to know if these spans are idle for no more than 5 minutes ?
Any help, thx

@Anteoy

This comment has been minimized.

Copy link
Author

commented Jul 31, 2019

if errno := madvise(v, n, int32(advise)); advise == _MADV_FREE && errno != 0 {
		// MADV_FREE was added in Linux 4.5. Fall back to MADV_DONTNEED if it is
		// not supported.
		atomic.Store(&adviseUnused, _MADV_DONTNEED)
		madvise(v, n, _MADV_DONTNEED)
	}

This code shows that my linux will only use MADV_DONTNEED.

@katiehockman

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2019

From #14521 (comment), it seems like you should be looking at HeapReleased, rather than HeapIdle. HeapReleased is increasing in your example, so it appears to be released to linux periodically as you would expect.

/cc @aclements @randall77 since they have more context here, and may better understand the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.