runtime,cmd/go: `runtime: out of memory` in `runtime.sysMapOS`

[bcmills]

#!watchflakes
post <- `runtime: out of memory` && `^\s*runtime\.sysMapOS`

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0xa25205?, 0x20000?})
	/tmp/workdir/go/src/runtime/panic.go:1047 +0x5d fp=0x243c4b248 sp=0x243c4b218 pc=0x43621d
runtime.sysMapOS(0xc001400000, 0xc000800000?)
	/tmp/workdir/go/src/runtime/mem_bsd.go:71 +0x11b fp=0x243c4b290 sp=0x243c4b248 pc=0x41975b
runtime.sysMap(0xe46c80?, 0x21b3a8000?, 0x42cb80?)
	/tmp/workdir/go/src/runtime/mem.go:142 +0x35 fp=0x243c4b2c0 sp=0x243c4b290 pc=0x419595
runtime.(*mheap).grow(0xe46c80, 0x6000292?)
	/tmp/workdir/go/src/runtime/mheap.go:1459 +0x23d fp=0x243c4b330 sp=0x243c4b2c0 pc=0x429c3d
runtime.(*mheap).allocSpan(0xe46c80, 0x6000292, 0x0, 0x1)
	/tmp/workdir/go/src/runtime/mheap.go:1191 +0x2e6 fp=0x243c4b3e0 sp=0x243c4b330 pc=0x4293c6
runtime.(*mheap).alloc.func1()
	/tmp/workdir/go/src/runtime/mheap.go:910 +0x65 fp=0x243c4b428 sp=0x243c4b3e0 pc=0x428d25
runtime.systemstack()
	/tmp/workdir/go/src/runtime/asm_amd64.s:492 +0x49 fp=0x243c4b430 sp=0x243c4b428 pc=0x464729

I notice that sysMapOS and sysReserveOS on BSD both invoke mmap, and cmd/go has recently started using syscall.Mmap to access cached metadata during package loading.

I can think of the following possible explanations, but I'm not sure which (if any) is correct.

1. Perhaps a bug in cmd/go (perhaps on the indexing path) is causing it to use more memory than expected?
2. Perhaps a bug in the OpenBSD mmap implementation causes the cmd/go call to syscall.Mmap to steal an address that was supposed to be reserved for the runtime?
3. Perhaps a bug somewhere in the runtime causes it to drastically overshoot the heap target? (But I would expect the syscall.Mmap usage to instead cause it to undershoot, since the heap size used to compute GOGC will not include the Mmap'd data.)

This is another case where it would be really helpful to have some heap stats from the runtime when the process OOMs (#52546)...

(CC @golang/openbsd @golang/runtime)

[bcmills]

Leaving on the backlog for now to see whether a pattern emerges. (Would be useful to know whether this failure mode is GOOS-specific.)

[costela]

I'm experiencing a similar issue while running go mod tidy in a project, which I suspect may be related:

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0xa4885c?, 0x0?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f371dffac80 sp=0x7f371dffac50 pc=0x436b7d
runtime.sysMapOS(0xc005000000, 0x400000?)
        /usr/local/go/src/runtime/mem_linux.go:187 +0x11b fp=0x7f371dffacc8 sp=0x7f371dffac80 pc=0x41999b
runtime.sysMap(0xc000000000?, 0xc3ffffffff?, 0x7f374d569000?)
        /usr/local/go/src/runtime/mem.go:142 +0x35 fp=0x7f371dffacf8 sp=0x7f371dffacc8 pc=0x419375
runtime.(*mheap).grow(0xe76b00, 0x4?)
        /usr/local/go/src/runtime/mheap.go:1522 +0x245 fp=0x7f371dffad70 sp=0x7f371dffacf8 pc=0x429aa5
runtime.(*mheap).allocSpan(0xe76b00, 0x4, 0x0, 0xef?)
        /usr/local/go/src/runtime/mheap.go:1243 +0x1b7 fp=0x7f371dffae08 sp=0x7f371dffad70 pc=0x4291f7
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:961 +0x65 fp=0x7f371dffae50 sp=0x7f371dffae08 pc=0x428ca5
runtime.systemstack()
        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7f371dffae58 sp=0x7f371dffae50 pc=0x466dc9

Of note: this is on linux, not BSD.

I haven't been able to find a minimal example yet (the project in question is unfortunately not small), but one interesting detail that might be interesting: cleaning the cache via go clean -cache allows the next go mod tidy to run normally.

This affected at least 1.19 and 1.20.

[pazof]

I'm experimenting a SIGSEGV, launching woodpecker server 1.15.7 from my Debian 11.6,
at the same source line : /usr/local/go/src/runtime/panic.go:1047 (having configured a Postgresql connection).

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- `runtime: out of memory` && `^\s*runtime\.sysMapOS`

2023-11-16 19:26 openbsd-386-72 go@d573a8a8 runtime/trace.TestTraceCPUProfile (log)

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x81ffb4e, 0x16})
	/tmp/workdir/go/src/runtime/panic.go:1023 +0x4d fp=0x12e44704 sp=0x12e446f0 pc=0x8082ddd
runtime.sysMapOS(0xad000000, 0x800000)
	/tmp/workdir/go/src/runtime/mem_bsd.go:81 +0x142 fp=0x12e44730 sp=0x12e44704 pc=0x805ef32
runtime.sysMap(0xad000000, 0x800000, 0x834b118)
	/tmp/workdir/go/src/runtime/mem.go:155 +0x3f fp=0x12e44740 sp=0x12e44730 pc=0x805ed0f
runtime.(*mheap).grow(0x833dde0, 0x375)
...
	/tmp/workdir/go/src/internal/trace/parser.go:269 +0x152b fp=0x6dcd8ba0 sp=0x6dcd8a6c pc=0x816775b
internal/trace.parse({0x8239b70, 0x6dde00f0}, {0x0, 0x0})
	/tmp/workdir/go/src/internal/trace/parser.go:104 +0x27 fp=0x6dcd8bec sp=0x6dcd8ba0 pc=0x8165f87
internal/trace.Parse({0x8239b70, 0x6dde00f0}, {0x0, 0x0})
	/tmp/workdir/go/src/internal/trace/parser.go:91 +0x37 fp=0x6dcd8c1c sp=0x6dcd8bec pc=0x8165e67
runtime/trace_test.parseTrace(0x6dca8300, {0x8239b70, 0x6dde00f0})
	/tmp/workdir/go/src/runtime/trace/trace_test.go:154 +0x47 fp=0x6dcd8d54 sp=0x6dcd8c1c pc=0x81b11c7
runtime/trace_test.TestTraceCPUProfile(0x6dca8300)
	/tmp/workdir/go/src/runtime/trace/trace_test.go:699 +0x461 fp=0x6dcd8f98 sp=0x6dcd8d54 pc=0x81b4361
testing.tRunner(0x6dca8300, 0x820a06c)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- `runtime: out of memory` && `^\s*runtime\.sysMapOS`

2023-11-20 15:56 openbsd-386-72 go@1d187fd0 runtime/trace (log)

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x8201b63, 0x16})
	/tmp/workdir/go/src/runtime/panic.go:1023 +0x4d fp=0x21321c84 sp=0x21321c70 pc=0x8082dfd
runtime.sysMapOS(0xae400000, 0x800000)
	/tmp/workdir/go/src/runtime/mem_bsd.go:81 +0x142 fp=0x21321cb0 sp=0x21321c84 pc=0x805ef12
runtime.sysMap(0xae400000, 0x800000, 0x834f118)
	/tmp/workdir/go/src/runtime/mem.go:155 +0x3f fp=0x21321cc0 sp=0x21321cb0 pc=0x805ecef
runtime.(*mheap).grow(0x8341de0, 0x375)
...
runtime.gopark(0x0, 0x0, 0xa, 0x10, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:402 +0xfa fp=0x6f4c2fd4 sp=0x6f4c2fc0 pc=0x8085dba
runtime.block()
	/tmp/workdir/go/src/runtime/select.go:103 +0x32 fp=0x6f4c2fe8 sp=0x6f4c2fd4 pc=0x80968b2
runtime/trace_test.TestTraceStressStartStop.func1.10()
	/tmp/workdir/go/src/runtime/trace/trace_test.go:496 +0x23 fp=0x6f4c2ff0 sp=0x6f4c2fe8 pc=0x81b6b73
runtime.goexit({})
	/tmp/workdir/go/src/runtime/asm_386.s:1363 +0x1 fp=0x6f4c2ff4 sp=0x6f4c2ff0 pc=0x80b77b1
created by runtime/trace_test.TestTraceStressStartStop.func1 in goroutine 35
	/tmp/workdir/go/src/runtime/trace/trace_test.go:494 +0x6c8

— watchflakes