Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: Memory not returned to system on linux #37585

Open
TimmyOVO opened this issue Feb 29, 2020 · 6 comments
Open

runtime: Memory not returned to system on linux #37585

TimmyOVO opened this issue Feb 29, 2020 · 6 comments

Comments

@TimmyOVO
Copy link

@TimmyOVO TimmyOVO commented Feb 29, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build842407928=/tmp/go-build -gno-record-gcc-switches"

What did you do?


import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
	}()
	time.Sleep(time.Minute * 20)
}

What did you expect to see?

memory should be returned to system

What did you see instead?

On macos the memory be returned to system after about 3 minutes, but after waiting for 20 minutes, the memory still being use by the application on linux
73A4582A-AECC-47C6-937E-28A89EADA0F9

@randall77
Copy link
Contributor

@randall77 randall77 commented Feb 29, 2020

Does using runtime/debug.FreeOSMemory help?
Does Go 1.14 help?

What method are you using to measure RSS? A common confusion is that while Go releases memory back to the OS, if the OS has no other demand for that memory it won't actually take the pages back. Only if there are other demands for that memory will the Go process's RSS go down.

Please don't post screenshots of code or logs. It makes it hard for us to reproduce your results. Just copy-paste the text directly.

@TimmyOVO
Copy link
Author

@TimmyOVO TimmyOVO commented Feb 29, 2020

update Golang to 1.14 doesn't help
using debug.FreeOSMemory either
but after downgrade Golang to 1.11 works

the code I'm using for test

package main

import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
		debug.FreeOSMemory()
	}()
	time.Sleep(time.Minute * 20)
}

Compiled by Golang 1.11.13

go version go1.11.13 darwin/amd64

A077DBDC-53E6-44B9-9A3C-585A3E1C0624

@thepudds
Copy link

@thepudds thepudds commented Feb 29, 2020

@TimmyOVO Are you on a relatively recent Linux?

If so, what does this show for the PID of your process:

grep LazyFree /proc/<PID>/smaps | grep -v " 0 kB"

From /proc man page:

LazyFree %lu (since Linux 4.12)
          Shows the amount of memory marked by madvise(2) MADV_FREE.

From Go 1.12 release notes:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed.

(On mobile, so brief).

@thepudds
Copy link

@thepudds thepudds commented Feb 29, 2020

I updated @TimmyOVO's code from above (#37585 (comment)) to periodically output RSS memory values as reported by ps and top, as well as to output the sum of LazyFree values from /proc/<pid>/smaps.

Full update is on playground, but the primary addition is this quick & dirty function:

reportMem := func(header string) {
	log.Println("---", header, "---")
	pid := os.Getpid()
	shell("ps rss:  ", `ps -o rss= -p %d`, pid)
	shell("top res: ", `top -b -n 1 | awk '/^[ ]*%d[ ]/ {print $6}'`, pid)
	shell("LazyFree:", `cat /proc/%d/smaps | awk '/LazyFree/ {sum+= $2} END {print sum}'`, pid)
}

I think the results are as expected for Go 1.11, 1.13, and 1.14, at least as far as I understand things in this general area:

  • In 1.11, RSS declines significantly after FreeOSMemory.
  • In 1.13 and 1.14, after 5 minutes RSS is still high while LazyFree has climbed to almost the same value as RSS (>300MB), and the subsequent FreeOSMemory does not seem to have material impact.

Go 1.11

$ go1.11.10 run main.go 
2020/02/29 22:33:01 --- before alloc ---
2020/02/29 22:33:01 ps rss:   1544
2020/02/29 22:33:02 top res:  1544
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:33:02 --- after alloc ---
2020/02/29 22:33:02 ps rss:   331076
2020/02/29 22:33:02 top res:  331076
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:38:02 --- after 5 minutes --- 
2020/02/29 22:38:02 ps rss:   331076
2020/02/29 22:38:02 top res:  331076
2020/02/29 22:38:02 LazyFree: 0
2020/02/29 22:38:02 --- after FreeOSMemory ---
2020/02/29 22:38:02 ps rss:   11908
2020/02/29 22:38:03 top res:  11908
2020/02/29 22:38:03 LazyFree: 0

Go 1.13

go1.13.3 run main.go
2020/02/29 22:40:05 --- before alloc ---
2020/02/29 22:40:05 ps rss:   1572
2020/02/29 22:40:06 top res:  1572
2020/02/29 22:40:06 LazyFree: 0
2020/02/29 22:40:06 --- after alloc ---
2020/02/29 22:40:06 ps rss:   331320
2020/02/29 22:40:06 top res:  331320
2020/02/29 22:40:06 LazyFree: 2384
2020/02/29 22:45:06 --- after 5 minutes ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319104
2020/02/29 22:45:06 --- after FreeOSMemory ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319032

Go 1.14

$ go1.14 run main.go 
2020/02/29 22:47:13 --- before alloc ---
2020/02/29 22:47:13 ps rss:   1852
2020/02/29 22:47:14 top res:  1852
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:47:14 --- after alloc ---
2020/02/29 22:47:14 ps rss:   331312
2020/02/29 22:47:14 top res:  331312
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:52:14 --- after 5 minutes ---
2020/02/29 22:52:14 ps rss:   331312
2020/02/29 22:52:14 top res:  331312
2020/02/29 22:52:14 LazyFree: 319352
2020/02/29 22:52:14 --- after FreeOSMemory ---
2020/02/29 22:52:14 ps rss:   331576
2020/02/29 22:52:14 top res:  331576
2020/02/29 22:52:14 LazyFree: 319372

This is on Ubuntu:

$ grep DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
$ uname -a
Linux instance-fzgo-test-7 4.15.0-1037-gcp #39-Ubuntu SMP Wed Jul 3 06:28:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
@dmitshur dmitshur added this to the Backlog milestone Mar 2, 2020
@dmitshur dmitshur added the OS-Linux label Mar 2, 2020
@bonedaddy
Copy link

@bonedaddy bonedaddy commented May 10, 2020

If you run your binary with GODEBUG=madvdontneed=1 memory will be returned to the OS

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 2, 2020

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux

gopherbot pushed a commit that referenced this issue Nov 2, 2020
In Go 1.12, we changed the runtime to use MADV_FREE when available on
Linux (falling back to MADV_DONTNEED) in CL 135395 to address issue
 #23687. While MADV_FREE is somewhat faster than MADV_DONTNEED, it
doesn't affect many of the statistics that MADV_DONTNEED does until
the memory is actually reclaimed under OS memory pressure. This
generally leads to poor user experience, like confusing stats in top
and other monitoring tools; and bad integration with management
systems that respond to memory usage.

We've seen numerous issues about this user experience, including
 #41818, #39295, #37585, #33376, and #30904, many questions on Go
mailing lists, and requests for mechanisms to change this behavior at
run-time, such as #40870. There are also issues that may be a result
of this, but root-causing it can be difficult, such as #41444 and
 #39174. And there's some evidence it may even be incompatible with
Android's process management in #37569.

This CL changes the default to prefer MADV_DONTNEED over MADV_FREE, to
favor user-friendliness and minimal surprise over performance. I think
it's become clear that Linux's implementation of MADV_FREE ultimately
doesn't meet our needs. We've also made many improvements to the
scavenger since Go 1.12. In particular, it is now far more prompt and
it is self-paced, so it will simply trickle memory back to the system
a little more slowly with this change. This can still be overridden by
setting GODEBUG=madvdontneed=0.

Fixes #42330 (meta-issue).

Fixes #41818, #39295, #37585, #33376, #30904 (many of which were
already closed as "working as intended").

Change-Id: Ib6aa7f2dc8419b32516cc5a5fc402faf576c92e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/267100
Trust: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.