Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abnormal memory usage on MIPS platform with GOMIPS=softfloat #39174

Open
Gh0u1L5 opened this issue May 20, 2020 · 5 comments
Open

Abnormal memory usage on MIPS platform with GOMIPS=softfloat #39174

Gh0u1L5 opened this issue May 20, 2020 · 5 comments
Milestone

Comments

@Gh0u1L5
Copy link

@Gh0u1L5 Gh0u1L5 commented May 20, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="mipsle"
GOBIN=""
GOCACHE="/home/build/.cache/go-build"
GOENV="/home/build/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/build/Go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.14"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.14/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
GOMIPS="softfloat"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -mabi=32 -march=mips32 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build072440335=/tmp/go-build -gno-record-gcc-switches"

What did you do?

  1. I compiled the following hello world program:
package main

import "fmt"

func main() {
    fmt.Println("Hello")
    for true {}
}
  1. I uploaded the compiled binary to my OpenWRT router, and executed the binary.

What did you expect to see?

The memory usage should be a reasonable number, e.g. 2~3MB.

What did you see instead?

The top program reported a crazy number (~650MB), and the program got killed by the OOM-killer later.

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 6339  1120 root     R     647m 541%  23% ./Hello

And More

So I've did some tests and got the following interesting observations:

  1. It has nothing to do with the kernel version. I've tried downgrade the kernel from 5.4.41 to 4.14.180, it makes no difference.

  2. I've tried GOGC=<some number>, it makes no difference.

  3. I've tried GODEBUG=madvdontneed=1 and GODEBUG=asyncpreemptoff=1 because I noticed that these features had caused some memory issues before. However, I got no luck on these two either.

  4. I haven't try GOARCH=mips or GOMIPS=hardfloat yet, because that OpenWRT router is the only MIPS device I have in my hand, and it doesn't support these options.

  5. I eventually found a temporary workaround: ulimit -v 32768, i.e. limit the memory usage to at most 32MB. After applying this limitation, I got a very reasonable memory usage:

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 6415  1120 root     R     6676   5%  25% ./Hello
  1. BTW, my router has only 120MB memory, thus this ~600MB memory usage makes absolutely no sense for me.
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented May 20, 2020

This is expected (VSZ reporting ~600 MiB), assuming that first line was generated on linux/amd64. Your issue title suggests that this is a problem on mips, but you talk about linux/amd64 earlier. Could you clarify which numbers were generated on which platforms, and where the OOM happened?

As of Go 1.14 the runtime initializes some structures by making large virtual mappings. Note that it does not actually ever map that memory unless it's needed (and how much is needed is proportional to your heap size), so Go 1.14 should not use much more physical memory than Go 1.13. If your process was OOM-killed it may be due to how your overcommit settings are configured (though in my experiments, I found that Linux really doesn't charge you for simply reserving address space, regardless of the overcommit setting), or it could be something else (a low ulimit for example). Are you positive your program was OOM-killed due to virtual memory usage?

I do not, in general, recommend using ulimit -v because these days virtual memory on most platforms is ~free, so virtual memory use is generally not a good proxy for limiting how much physical memory a process can use. 32-bit platforms are an exception, but we don't make such large virtual mappings on 32-bit. If you're seeing large mappings on 32-bit, that's a problem.

@mknyszek mknyszek added this to the Go1.15 milestone May 20, 2020
@Gh0u1L5
Copy link
Author

@Gh0u1L5 Gh0u1L5 commented May 20, 2020

So, what I'm doing is cross compiling a binary for a MIPS device (a OpenWRT router). If you check the go env output, you will notice that the GOHOSTARCH is amd64, and GOARCH is mipsle. The OOM happens when I execute the compiled binary on MIPS platform.

The explanation about virtual mappings makes sense for me. I double checked the log, and noticed that the hello world program actually never get killed by the OOM-killer. The one get killed is the program I'm actually working on, a traffic forwarding program written in Go.

So, I guess what is actually happening is, the traffic forwarding program can see a large trunk of virtual memory (~650MB), without knowing that the physical memory is merely 120MB. So it keeps charging new memory without triggering GC. Eventually, it hits the hard limit, OOM-killer jumps up, crash. Does this guess make sense to you?

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented May 21, 2020

So, what I'm doing is cross compiling a binary for a MIPS device (a OpenWRT router). If you check the go env output, you will notice that the GOHOSTARCH is amd64, and GOARCH is mipsle. The OOM happens when I execute the compiled binary on MIPS platform.

I see... then it is somewhat surprising to me that there's such a large mapping on mipsle, which is 32-bit. There may be a real issue here since there isn't much address space on 32-bit.

The explanation about virtual mappings makes sense for me. I double checked the log, and noticed that the hello world program actually never get killed by the OOM-killer. The one get killed is the program I'm actually working on, a traffic forwarding program written in Go.

Got it, thank you for the clarification.

So, I guess what is actually happening is, the traffic forwarding program can see a large trunk of virtual memory (~650MB), without knowing that the physical memory is merely 120MB. So it keeps charging new memory without triggering GC. Eventually, it hits the hard limit, OOM-killer jumps up, crash. Does this guess make sense to you?

I'm not sure I follow. Does the traffic forwarding program explicitly reference the virtual memory usage for some kind of throttling mechanism? And I'm not sure what you mean by triggering a GC, either; if you have GOGC set to e.g. 100, then GCs will trigger when the heap reaches double the live heap at the end of the last GC (which is all tracked by updating metadata when allocations are made; it doesn't read any OS-reported statistics). It's not triggered based on a memory maximum or anything.

@Gh0u1L5
Copy link
Author

@Gh0u1L5 Gh0u1L5 commented May 21, 2020

So, I guess what is actually happening is, the traffic forwarding program can see a large trunk of virtual memory (~650MB), without knowing that the physical memory is merely 120MB. So it keeps charging new memory without triggering GC. Eventually, it hits the hard limit, OOM-killer jumps up, crash. Does this guess make sense to you?

I'm not sure I follow. Does the traffic forwarding program explicitly reference the virtual memory usage for some kind of throttling mechanism? And I'm not sure what you mean by triggering a GC, either; if you have GOGC set to e.g. 100, then GCs will trigger when the heap reaches double the live heap at the end of the last GC (which is all tracked by updating metadata when allocations are made; it doesn't read any OS-reported statistics). It's not triggered based on a memory maximum or anything.

Ummm? Interesting. I didn't know that the GC in Go is triggered by the increasing rate. Then how does ulimit help relieve my issue?

@Gh0u1L5
Copy link
Author

@Gh0u1L5 Gh0u1L5 commented May 23, 2020

Update: I thought the ulimit settings help relieve the issue. But it turns out that what really helped are some iptable rules set by another team. These iptable rules redirects a lot of connections to another router, which means the traffic forwarding program now experiences far less pressure. So I guess the OOM issue is mainly caused by poor memory optimization of the program itself, I'll spend some time working on that.

In the mean while, I'll keep the issue open until you figure out whether 650MB virtual memory is normal for mipsle or not. Good luck and have a nice weekend!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.