Skip to content

runtime: large address space footprint on 32-bit linux #35677

Open
@tmm1

Description

@tmm1

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/arm

Summary

Running a simple golang program on linux/arm (or other 32-bit linux arch) shows ~800MB of the virtual address space is reserved up front:

package main

import (
	"fmt"
	"os"
	"os/exec"
)

func main() {
	cmd := exec.Command("pmap", "-x", fmt.Sprint(os.Getpid()))
	cmd.Stdout = os.Stdout
	cmd.Run()
}
8627:   /tmp/testpmap
Address   Kbytes     RSS   Dirty Mode  Mapping
00010000     624     444     444 r-x-- testpmap
000b0000     704     176     176 r---- testpmap
00160000      72      36      36 rw--- testpmap
00172000      68      28      28 rw---   [ anon ]
01400000    4096     196     196 rw---   [ anon ]
01800000  524288       0       0 -----   [ anon ]
66cb4000     644      52      52 rw---   [ anon ]
66d55000  264060       0       0 -----   [ anon ]
76f34000     256      32      32 rw---   [ anon ]
76f74000       4       0       0 r-x--   [ anon ]
7ee2c000     132      12      12 rw---   [ stack ]
ffff0000       4       0       0 r-x--   [ anon ]
-------- ------- ------- -------
total kB  794952     976     976

The Linux kernel can be compiled with one of three different vmsplit modes: VMSPLIT_3G, VMSPLIT_2G, VMSPLIT_1G


I have a golang program that makes heavy use of memory mapped files, and runs on ARM appliances where the vendor has compiled their kernel with VMSPLIT_2G. Since the golang runtime reserves 40% of the available address space up-front, my program is limited in the amount of files it can mmap for its own purposes. When the address space is exhausted, bad things happen including panics when trying to spawn threads:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0xb68bd6 m=2 sigcode=4294967290

The golang runtime documents its memory mappings in malloc.go. The ~800MB reserved is made up of two large reservations: the first is ~258MB and the second is 512MB.

go/src/runtime/malloc.go

Lines 541 to 545 in a23f9af

// 1. We reserve space for all heapArenas up front so
// they don't get interleaved with the heap. They're
// ~258MB, so this isn't too bad. (We could reserve a
// smaller amount of space up front if this is a
// problem.)

go/src/runtime/malloc.go

Lines 551 to 552 in a23f9af

// 3. We try to stake out a reasonably large initial
// heap reservation.


The 512MB initial reservation can be tweaked with a patch to arenaSizes:

go/src/runtime/malloc.go

Lines 586 to 592 in a23f9af

arenaSizes := []uintptr{
512 << 20,
256 << 20,
128 << 20,
}
for _, arenaSize := range arenaSizes {
a, size := sysReserveAligned(unsafe.Pointer(p), arenaSize, heapArenaBytes)

However, it is not clear to me how I can reduce the ~258MB reservation. The code comment states "We could reserve a smaller amount of space up front if this is problem.", so I'm looking for some guidance on how to do this.

Since userspace on 32-bit linux can access a maximum of 3GB of memory (when the kernel is compiled with VMSPLIT_3G [which is the default]), it seems like at the very least the 258MB reservation could be reduced by 1/4 to ~198MB.

cc @aclements

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.compiler/runtimeIssues related to the Go compiler and/or runtime.

    Type

    No type

    Projects

    Status

    Triage Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions