runtime: helloworld net/http asks 700MB VSS on mips32

[laoshaw]

What version of Go are you using (go version)?

go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Not sure

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lshaw/.cache/go-build"
GOENV="/home/lshaw/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/lshaw/.gvm/pkgsets/go1.15/global/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lshaw/.gvm/pkgsets/go1.15/global"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/lshaw/.gvm/gos/go1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/lshaw/.gvm/gos/go1.15/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build821697598=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I build a helloworld net/http using : GOOS=linux GOARCH=mips GOMIPS=softfloat go build -ldflags="-s -w" and run it on a mips32 board, use top to notice its VSS is 700MB and RSS is 4MB. run multiple of them in parallel will cause 'out of memory' when I set overcommit_memory to 2 very quickly. I have 128MB RAM

The code I use:

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"strconv"
	"time"
)

func main() {
	min := 8080
	max := 60000

	s1 := rand.NewSource(time.Now().UnixNano())
	r1 := rand.New(s1)

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Fosiao you had youre request: %s\n", r.URL.Path)
	})

	port := ":" + strconv.Itoa(r1.Intn(max-min))
	//fmt.Println(port)
	http.ListenAndServe(port, nil)
}

What did you expect to see?

I would expect a much smaller VSS on a 128MB 32-bit system, to compare my C/C++ http process only need about 4MB for VSS and 1MB for RSS

What did you see instead?

700MB VSS and easily 'out of memory' errors.

[mknyszek]

Unfortunately I got caught up with a few things today but I'll take a look tomorrow if I can grab our MIPS builder (shouldn't be a problem, CI is pretty quiet this close to release).

[mknyszek]

Also this sound suspiciously familiar. The GOARCH=mips GOMIPS=softfloat is reminding me of #39174, which was closed before I ended up actually looking into it. That makes me think there actually is a bug here and somehow the runtime thinks 32-bit mips has a much larger address space than it actually does.

I just quickly did some of the math for 32-bit mips by hand in case there's some crazy overflow happening here or something, since technically we treat 32-bit mips as having a 31-bit address space.

The math seems to work out though, at least for the page allocator. It comes out to around 4680 bytes (then rounded up a physical page in size) for the summary structure, plus another 32 KiB for the page bitmap. It's still possible the page allocator is the problem, there could still be a bug or it's possible I made a mistake. Best thing would be to just run this on the right hardware with some extra print statements to understand what's going on.

[laoshaw]

if you have some testing code with the print embedded I'm happily to run them and send you the output, I have the hardware nearby. Thanks for looking into this quickly.

[mknyszek]

Alrighty, so I dug into this and the large reservation appears to be WAI. The reservation is not from the changes in Go 1.14; this code is quite old. (And silly me, I actually knew we did this; I had to update the relevant code some time ago, but the 600 MiB total really threw me off.)

On 32-bit platforms there are two big up-front reservations. First, we attempt to make a large reservation up-front for heap bookkeeping data structures. Then we make a large reservation for the heap itself because we're concerned about fragmenting the address space (see https://cs.opensource.google/go/go/+/master:src/runtime/malloc.go;l=551;bpv=1).

Note that this reservation is strictly PROT_NONE and pieces are mapped in as read-write gradually, only as needed. Linux overcommit should be ignoring the parts of the reservation that are never touched.

So that solves the puzzle of where the VSS is coming from. But the other question is why you're running into overcommit issues. I suspect that this may be due to some heap bookkeeping data structures which are not yet used (and thus don't appear in RSS) but are committed as read-write. For instance, there's a bitmap that accounts for about 3% overhead of the peak heap usage, and this is all memory that stays committed.

But that doesn't square with some of the values I'm seeing. On a real linux/mips machine I printed out the sizes passed to every mmap call we make:

sysAlloc 262144
sysReserve 8192
sysMap 8192
sysReserve 135366656
sysReserve 541065216
sysMap 4194304
sysMap 266240
sysAlloc 65536
sysAlloc 65536
sysAlloc 65536
sysAlloc 262144

sysReserve lines are strictly PROT_NONE mappings, sysMap are applications of PROT_READ|PROT_WRITE to existing PROT_NONE mappings, and sysAlloc is a new mapping that's PROT_READ|PROT_WRITE.

The total amount of committed memory is thus 5189632 bytes, or just about 5 MiB. The physical page size on this machine is 8 KiB. If the page size is higher on your machine, these numbers will likely be slightly higher (round all of them up to that size).

You mentioned in the golang-nuts thread running about 40 of these simultaneously, correct? By my calculations, the most you could run if you weren't using any other memory on the system would be 25 before overcommit fired. It's also not super surprising to me that overcommit fires first in your case because a lot of this memory (including that 4 MiB read-write mapping, which is the first heap arena) hasn't been touched. The application is otherwise sitting idle, so it's not necessarily using all the space allocated for the various data structures, and it may not have used the whole heap, either.

So, that's my best analysis of the situation.

Regarding the large VSS usage, that's WAI and a problem for ulimit -v, but the code has worked like that for a very long time. We could talk about reducing the up-front reservations, but it won't help your immediate problem with overcommit.

Your immediate problem can likely only be remedied by having a smaller minimum heap size (the current is 4 MiB) and mapping in the heap more incrementally. A smaller minimum heap size is blocked hard on #42430 which I'm actively investing time into resolving, but will take time to complete.

[mknyszek]

Because addressing this better depends somewhat on #44167 (which should hopefully enable a smaller heap minimum), and I had to push that back, I need to push this back as well. My apologies.

[mknyszek]

Putting this into the backlog.

Though, the minimum heap size for 1.18 has been decreased to 512 KiB with #44167. The heap is still mapped in 4 MiB increments however, so I'm not sure this helps much with the overcommit issue.