runtime: high startup memory allocation on Linux AMD64

[pkramme]

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/vorvvbgc/.cache/go-build"
GOENV="/home/vorvvbgc/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vorvvbgc/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/vorvvbgc/go1.14/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vorvvbgc/go1.14/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build969828813=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am trying to get an FastCGI server running behind an Apache2 webserver on a shared hosting system using the net/http/fcgi library. The webserver is limiting my software to 512MB memory.

This is the code: https://play.golang.org/p/Z-Gc6icOpw5

What did you expect to see?

I expect to see "This was generated by Go running as a FastCGI app" on the website generated by the FastCGI server.

What did you see instead?

I have modified the sysReserve() function in the runtime to include println() to print out the error code from mmap() and the requested memory size. This is a diff of src/runtime/mem_linux.go and my version:

157a158,159
>       println(err)
>       println(n)

I kept the output in the following output in the hopes that it might be useful.

The application crashes with this trace:

0
131072
0
1048576
0
8388608
0
67108864
12
536870912
fatal error: failed to reserve page summary memory

runtime stack:
runtime.throw(0x6f3456, 0x25)
        /home/vorvvbgc/go1.14/go/src/runtime/panic.go:1112 +0x72 fp=0x7ffc17e5b170 sp=0x7ffc17e5b140 pc=0x433a12
runtime.(*pageAlloc).sysInit(0x939428)
        /home/vorvvbgc/go1.14/go/src/runtime/mpagealloc_64bit.go:80 +0x13f fp=0x7ffc17e5b1e8 sp=0x7ffc17e5b170 pc=0x42ac1f
runtime.(*pageAlloc).init(0x939428, 0x939420, 0x94db38)
        /home/vorvvbgc/go1.14/go/src/runtime/mpagealloc.go:297 +0x75 fp=0x7ffc17e5b210 sp=0x7ffc17e5b1e8 pc=0x4288b5
runtime.(*mheap).init(0x939420)
        /home/vorvvbgc/go1.14/go/src/runtime/mheap.go:694 +0x274 fp=0x7ffc17e5b238 sp=0x7ffc17e5b210 pc=0x425ad4
runtime.mallocinit()
        /home/vorvvbgc/go1.14/go/src/runtime/malloc.go:470 +0xff fp=0x7ffc17e5b268 sp=0x7ffc17e5b238 pc=0x40c41f
runtime.schedinit()
        /home/vorvvbgc/go1.14/go/src/runtime/proc.go:545 +0x60 fp=0x7ffc17e5b2c0 sp=0x7ffc17e5b268 pc=0x437100
runtime.rt0_go(0x7ffc17e5b2f8, 0x1, 0x7ffc17e5b2f8, 0x0, 0x7fd2029790ca, 0x1, 0x7ffc17e5cbb6, 0x0, 0x7ffc17e5cbc4, 0x7ffc17e5cbe6, ...)
        /home/vorvvbgc/go1.14/go/src/runtime/asm_amd64.s:214 +0x125 fp=0x7ffc17e5b2c8 sp=0x7ffc17e5b2c0 pc=0x460655

The application works fine with golang 1.13.9.

I have no idea how to debug this further.

[andybons]

@aclements

[ivzhh]

@pkramme Hi, would you mind to post your Apache setup for this too? I did not reproduce this on a fresh Apache. Maybe it is due to my configuration.

[alexzorin]

@pkramme is this shared hosting environment cPanel by any chance?

We also started getting reports of this same panic with our Go application, which exposes itself as a .live.cgi FastCGI net/http/cgi server integrating with cPanel's LiveAPI, as soon as we upgraded to 1.14.

Going to downgrade to 1.13.9 for now.

[pkramme]

@aleksator No, it is not, it is a custom build setup. @ivzhh I'm not able to share the config, as it is proprietary.

Theoretically, if we execute any code with 512MB memory limitation, the problem should become visible. I will try to produce something not based on fcgi as a reproducer, so that no apache2 setup is necessary.

[aleksator]

Tagging a proper person here: @alexzorin

[alexzorin]

I think what @pkramme suggested about the 512MB memory limit is correct - specifically RLIMIT_AS.

"Back in the day" (EL5-ish era), shared web hosting admins did not have access to the RSS cgroups controller (because of EL5's ancient kernel), and so controlling VSZ limits was the only choice available to them. In the long term, this has resulted in a lot of misguided admins keeping these VSZ limits around for no good reason.

Anyway, the Apache-based reproducer is straightforward. (For some reason, a simple Go hello world wrapped in a bash ulimit -v didn't repro for me, not sure why).

1. Compile a very simple net/http/cgi binary using Go 1.14.1 and stick it in Apache httpd 2.4's cgi-bin/:

package main

import (
	"net/http/cgi"
)

func main() {
	if err := cgi.Serve(nil); err != nil {
		panic(err)
	}
}

go build -o /var/www/html/cgi-bin/reproducer.cgi reproducer.go

2. Configure Apache with a 512MB RLimitMEM and restart Apache (note, don't try this in Docker or LXC-like environments, setrlimit will just fail and the repro won't work):

RLimitMEM 536870912

apachectl -k restart

3. Access http://localhost/cgi-bin/reproducer.cgi. It will produce an HTTP 500, and in Apache's error_log, you will see the panic stack from the original report.

I would prefer not to ask our customers to remove the rlimit (or else we'll be stuck shipping with Go 1.13 for all eternity).

Would it be practical for the Go runtime to try work within whatever it sees by getrlimit?