Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GODEBUG = madvdontneed = 1 does not seem to take effect #39295

Closed
sxfworks opened this issue May 28, 2020 · 9 comments
Closed

GODEBUG = madvdontneed = 1 does not seem to take effect #39295

sxfworks opened this issue May 28, 2020 · 9 comments

Comments

@sxfworks
Copy link

@sxfworks sxfworks commented May 28, 2020

What version of Go are you using (go version)?

$ go version
go version go1.10.1 linux/amd64
go version go1.12.4 linux/amd64
go version go1.13.4 linux/amd64
go version go1.14 linux/amd64
go version go1.14.3 linux/amd64

Does this issue reproduce with the latest release?

yes,it still exists

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build216433097=/tmp/go-build -gno-record-gcc-switches"

What did you do?

What did you expect to see?

The same copy of golang code is compiled with go1.10 / go1.11, and the running memory is relatively stable. After the flow pressure is stabilized, the memory drops slightly, and after compiling with go1.12 / go1.13 / go1.14, the process memory is after the sudden burst of traffic pressure, it has been going up. If you do this repeatedly, it will lead to the process because of OOM is killed, compared with the golang version, it is found that MADV_FREE was introduced after go1.12, so use GODEBUG = madvdontneed = 1 ./bin way to start the program, multiple tests still did not alleviate this problem.

What did you see instead?

@davecheney
Copy link
Contributor

@davecheney davecheney commented May 28, 2020

The GODEBUG variable is easy to get wrong, there is little validation provided so the exact invocation is critical. Can you please

  1. Provide the exact steps you used so that someone without access to your environment can reproduce them. At a minimum a screenshot, or preferably the text of the commands that you used, should be provided.
  2. How are you determining memory usage. Please be as specific as possible.
@sxfworks
Copy link
Author

@sxfworks sxfworks commented May 28, 2020

1.I mainly control the start, stop and restart specific service file contents of the program through systemctl as follows:

⁣[Unit]
Description=exam
After=network.target

[Service]
User=exam
Group=exam
Type=simple
ExecStartPre="ulimit -c unlimited"
Envirment="GODEBUG=madvdontneed=1"
ExecStart=/opt/bin/binary_name
PrivateTmp=true
Restart=always

[Install]
WantedBy=multi-user.target

⁣2.I determine the memory usage rate, currently mainly through the following means:
(1) VIRT and RES in top. In the version of go1.12+, VIRT and RES are always going up and not going down
(2) Print some content in runtime.MemStats to metrics for display, the specific display is as follows:
image
image
image
image
⁣(3) Memory monitoring of the whole system
image

@mdlayher
Copy link
Member

@mdlayher mdlayher commented May 28, 2020

Envirment=

I see a typo in your systemd config. What happens if you fix this?

@sxfworks
Copy link
Author

@sxfworks sxfworks commented May 28, 2020

Envirment=

I see a typo in your systemd config. What happens if you fix this?

When I use go1.12 + or higher for stress testing, the golang heap does not return to the system. Repeated stress testing will lead to OOM. I have consulted that by setting GODEBUG = madvdontneed = 1, the GC can be adjusted to return the memory To the operating system, so this environment variable is set

@sxfworks
Copy link
Author

@sxfworks sxfworks commented May 28, 2020

When I use go1.10 / go1.11, the golang heap will be returned to the operating system, and will not be killed by the Linux system because of OOM

@mdlayher
Copy link
Member

@mdlayher mdlayher commented May 28, 2020

I'm not sure you understand my point: "Envirment" is a misspelling of "Environment" in your systemd unit. You should fix the typo in the unit, reload the configuration, and then report back if the issue persists.

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 2, 2020

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux

gopherbot pushed a commit that referenced this issue Nov 2, 2020
In Go 1.12, we changed the runtime to use MADV_FREE when available on
Linux (falling back to MADV_DONTNEED) in CL 135395 to address issue
 #23687. While MADV_FREE is somewhat faster than MADV_DONTNEED, it
doesn't affect many of the statistics that MADV_DONTNEED does until
the memory is actually reclaimed under OS memory pressure. This
generally leads to poor user experience, like confusing stats in top
and other monitoring tools; and bad integration with management
systems that respond to memory usage.

We've seen numerous issues about this user experience, including
 #41818, #39295, #37585, #33376, and #30904, many questions on Go
mailing lists, and requests for mechanisms to change this behavior at
run-time, such as #40870. There are also issues that may be a result
of this, but root-causing it can be difficult, such as #41444 and
 #39174. And there's some evidence it may even be incompatible with
Android's process management in #37569.

This CL changes the default to prefer MADV_DONTNEED over MADV_FREE, to
favor user-friendliness and minimal surprise over performance. I think
it's become clear that Linux's implementation of MADV_FREE ultimately
doesn't meet our needs. We've also made many improvements to the
scavenger since Go 1.12. In particular, it is now far more prompt and
it is self-paced, so it will simply trickle memory back to the system
a little more slowly with this change. This can still be overridden by
setting GODEBUG=madvdontneed=0.

Fixes #42330 (meta-issue).

Fixes #41818, #39295, #37585, #33376, #30904 (many of which were
already closed as "working as intended").

Change-Id: Ib6aa7f2dc8419b32516cc5a5fc402faf576c92e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/267100
Trust: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@doudouwyh
Copy link

@doudouwyh doudouwyh commented Nov 21, 2020

Hi, every one!
i use this to run my server, but rss still very high
GODEBUG=madvdontneed=1 ./my_server

pls tell me why, and how to use madvdontneed, thx!

go version is 1.14.2
centos7
amd64

@mdlayher
Copy link
Member

@mdlayher mdlayher commented Nov 21, 2020

@golang golang locked as resolved and limited conversation to collaborators Nov 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.