New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: programs compiled by 1.11 allocate an unreasonable amount of virtual memory #28114

Open
HouzuoGuo opened this Issue Oct 10, 2018 · 9 comments

Comments

Projects
None yet
7 participants
@HouzuoGuo

HouzuoGuo commented Oct 10, 2018

What version of Go are you using (go version)?

go1.11.1 linux/amd6

Does this issue reproduce with the latest release?

It was first discovered under go 1.11, and the issue remains in the minor upgrade to 1.11.1.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/howard/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/howard/gopath"
GOPROXY=""
GORACE=""
GOROOT="/home/howard/go"
GOTMPDIR=""
GOTOOLDIR="/home/howard/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build718493987=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compile the following trivial program:

package main

func main() {
        for {
        }
}

go build -o usagetest; ./usagetest and observe its memory usage as monitored by host operating system, for example via ps aux | grep usagetest.

What did you expect to see?

Compiled by 1.10.4, the program uses a very reasonable amount of virtual (2MB) and resident (600KB) memory:

howard 10838 101 0.0 2384 688 pts/2 Rl+ 05:57 0:59 ./usagetest

But when compiled by 1.11 (or 1.11.1), the virtual memory usage dramatically increases to 102MB:

howard 11326 99.5 0.1 101820 4776 pts/2 Rl+ 05:59 1:58 ./usagetest

The significant increase in virtual memory usage is usually not an issue, however security sensitive programs often lock their memory, causing a far greater performance degradation on low-spec computer hosts.

@randall77

This comment has been minimized.

Contributor

randall77 commented Oct 10, 2018

Related: #28081 (large core files)

@ianlancetaylor ianlancetaylor changed the title from Programs compiled by 1.11 allocate an unreasonable amount of virtual memory to runtime: programs compiled by 1.11 allocate an unreasonable amount of virtual memory Oct 10, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Oct 10, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 10, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 10, 2018

Presumably related to https://golang.org/cl/85888 and #10460.

@aclements

This comment has been minimized.

Member

aclements commented Oct 10, 2018

Interesting. I had thought that mlocking memory would only lock pages from being paged out, but wouldn't eagerly fault the pages in. But looking more closely at the manpage, it looks like that's only true with mlock2 with the MLOCK_ONFAULT flag or mlockall with the MCL_ONFAULT flag. I assume you're using mlockall? MCL_ONFAULT has only been available since Linux 4.4, but would that be a reasonable workaround, at least for now?

The solution I proposed in #28081 should also fix this.

@networkimprov

This comment has been minimized.

networkimprov commented Oct 10, 2018

@HouzuoGuo

This comment has been minimized.

HouzuoGuo commented Oct 11, 2018

Thanks for the hint @aclements.

Indeed, couple of my system programs are using mlockall, often with both syscall.MCL_CURRENT | syscall.MCL_FUTURE. For this issue report, MCL_CURRENT alone is sufficient:

package main

import "syscall"

func main() {
        if err := syscall.Mlockall(syscall.MCL_CURRENT); err != nil {
                panic(err)
        }
        for {
        }
}

As observed by host OS, it appears to have indeed copied all of the allocated virtual memory into main memory:

root 29281 156 2.5 101836 101736 pts/0 RLl+ 05:27 0:01 ./usagetest

Hehe also I was wondering why this issue attracted so many emojis, it turns out someone put it on hackernews..

@kumarharsh

This comment has been minimized.

kumarharsh commented Oct 11, 2018

Yeah, I had just upgraded to 1.11 while not noticing anything significant in the release notes (at that time). And just after that, I opened hacker news, and this was the top result. Imagine the panic! (pun intended)

@aclements

This comment has been minimized.

Member

aclements commented Oct 11, 2018

@HouzuoGuo, yes, that's what I would expect. Thanks for confirming.

I don't really understand when it makes sense to use mlockall instead of mlock. There's a ton of stuff in memory that certainly doesn't have to be locked, and using mlock for just the things that need to be locked wouldn't have the problem with virtual memory. What's the reason for using mlockall instead of mlock? And if you do need mlockall, is there a reason not to use MCL_ONFAULT?

Even if we do modify arena mapping to be incremental (which I believe should be relatively easy to do), I don't want to change how the arena index is mapped because that would impact both performance and simplicity. So mlockall would still fault in about 32MB, even with the change to arena mapping.

@HouzuoGuo

This comment has been minimized.

HouzuoGuo commented Oct 12, 2018

Thanks @aclements , TIL about MCL_ONFAULT, apparently available since Linux 4.4. I shall use it from now and onward.

Beyond the scope of this issue report, would you please offer some hints on the proper invocation of mlock in protecting a sensitive instance of structure, especially to determine its memory address range? Take this structure for example

type Daemon struct {
    ListenAddress string
    ListenPort int
    InternalData map[string]interface{}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment