runtime: sigpanic during GC on android/arm64

[tmm1]

What version of Go are you using (go version)?

go1.9.1

What operating system and processor architecture are you using (go env)?

android/arm64

What did you do?

I have a large golang server application that uses a variety of libraries, including some that use cgo. The app runs on many different platforms and is deployed on hundreds of different hardware and operating systems combinations.

On android/arm64 specifically, I see periodic crashes in the golang runtime.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4000017698 pc=0x20285915f8]

runtime stack:
runtime.throw(0x2028bda477, 0x2a)
	go/src/runtime/panic.go:605 +0x70
runtime.sigpanic()
	go/src/runtime/signal_unix.go:351 +0x264
runtime.inheap(...)
	go/src/runtime/mheap.go:377
runtime.gcmarkwb_m(0x442052bf20, 0x4425da67f0)
	go/src/runtime/mbarrier.go:163 +0xa0
runtime.writebarrierptr_prewrite1.func1()
	go/src/runtime/mbarrier.go:193 +0x54
runtime.systemstack(0x4420524f38)
	go/src/runtime/asm_arm64.s:241 +0x8c
runtime.mstart()
	go/src/runtime/proc.go:1125

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4000006578 pc=0x203b7dd330]

runtime stack:
runtime.throw(0x203be25477, 0x2a)
	go/src/runtime/panic.go:605 +0x70
runtime.sigpanic()
	go/src/runtime/signal_unix.go:351 +0x264
runtime.heapBitsForObject(0x442195e660, 0x442195b770, 0x0, 0x441ff361cb, 0x4400000000, 0x204da725f0, 0x442001d260, 0x68)
	go/src/runtime/mbitmap.go:392 +0x88
runtime.scanobject(0x442195b770, 0x442001d260)
	go/src/runtime/mgcmark.go:1187 +0x218
runtime.gcDrain(0x442001d260, 0xd)
	go/src/runtime/mgcmark.go:943 +0x1dc
runtime.gcBgMarkWorker.func2()
	go/src/runtime/mgc.go:1796 +0x16c
runtime.systemstack(0x442001e600)
	go/src/runtime/asm_arm64.s:241 +0x8c
runtime.mstart()
	go/src/runtime/proc.go:1125

I also observed this issue before upgrading to golang 1.9. Here's a crash from golang 1.8:

unexpected fault address 0x400004bea8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x400004bea8 pc=0x200cfc127c]

goroutine 5 [running]:
runtime.throw(0x200d4f970a, 0x5)
	go/src/runtime/panic.go:596 +0x70 fp=0x44207ad530 sp=0x44207ad510
runtime.sigpanic()
	go/src/runtime/signal_unix.go:297 +0x224 fp=0x44207ad580 sp=0x44207ad530
runtime.bulkBarrierPreWrite(0x4432faaea8, 0x4432f44a28, 0x10)
	go/src/runtime/mbitmap.go:581 +0x324 fp=0x44207ad610 sp=0x44207ad590
runtime.typedmemmove(0x200d6650c0, 0x4432faaea8, 0x4432f44a28)
	go/src/runtime/mbarrier.go:237 +0xa4 fp=0x44207ad640 sp=0x44207ad610
runtime.evacuate(0x200d6aeac0, 0x4426172570, 0x2)
	go/src/runtime/hashmap.go:1072 +0x5b4 fp=0x44207ad730 sp=0x44207ad640
runtime.growWork(0x200d6aeac0, 0x4426172570, 0x1)
	go/src/runtime/hashmap.go:957 +0x8c fp=0x44207ad750 sp=0x44207ad730
runtime.mapassign(0x200d6aeac0, 0x4426172570, 0x44207ad8a0, 0x200dc5b5d8)
	go/src/runtime/hashmap.go:513 +0x524 fp=0x44207ad7f0 sp=0x44207ad750

Slightly longer versions of these crashes are available in blevesearch/bleve#634, but I'm also happy to provide the full list of goroutine backtraces if that is helpful.

Given that the same codebase works fine on linux/freebsd/windows across amd64/i686/arm, I suspect this issue is specific to the arm64 golang runtime.

cc @aclements

[tmm1]

Looks like all the crashes happen when indexing into mheap_.spans. Both inheap and heapBitsForObject perform bounds checking before reaching into spans, so I'm not sure why it's still failing.

Is this some sort of memory corruption?

go/src/runtime/mheap.go

Lines 372 to 377 in 93322a5

	func inheap(b uintptr) bool {
	if b == 0 || b < mheap_.arena_start || b >= mheap_.arena_used {
	return false
	}
	// Not a beginning of a block, consult span table to find the block beginning.
	s := mheap_.spans[(b-mheap_.arena_start)>>_PageShift]

go/src/runtime/mbitmap.go

Lines 383 to 392 in 93322a5

	func heapBitsForObject(p, refBase, refOff uintptr) (base uintptr, hbits heapBits, s *mspan, objIndex uintptr) {
	arenaStart := mheap_.arena_start
	if p < arenaStart || p >= mheap_.arena_used {
	return
	}
	off := p - arenaStart
	idx := off >> _PageShift
	// p points into the heap, but possibly to the middle of an object.
	// Consult the span table to find the block beginning.
	s = mheap_.spans[idx]

[tmm1]

I found another crash report from golang1.8 which is slightly more interesting, but ultimately also happening when accessing heap spans.

runtime.(*mheap).allocSpanLocked

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x400005c0d8 pc=0x2026fc398c]

runtime stack:
runtime.throw(0x202750aaca, 0x2a)
        go/src/runtime/panic.go:596 +0x70
runtime.sigpanic()
        go/src/runtime/signal_unix.go:274 +0x26c
runtime.(*mheap).allocSpanLocked(0x2027c3c4a0, 0x1, 0x2027fa8860)
        go/src/runtime/mheap.go:726 +0xd4
runtime.(*mheap).alloc_m(0x2027c3c4a0, 0x1, 0x1e, 0x0)
        go/src/runtime/mheap.go:562 +0xc4
runtime.(*mheap).alloc.func1()
        go/src/runtime/mheap.go:627 +0x40
runtime.systemstack(0x7ff89d8b88)
        go/src/runtime/asm_arm64.s:255 +0xb8
runtime.(*mheap).alloc(0x2027c3c4a0, 0x1, 0x1000000001e, 0x2026fb705c)
        go/src/runtime/mheap.go:628 +0x60
runtime.(*mcentral).grow(0x2027c3e130, 0x0)
        go/src/runtime/mcentral.go:212 +0x78
runtime.(*mcentral).cacheSpan(0x2027c3e130, 0x4)
        go/src/runtime/mcentral.go:93 +0x350
runtime.(*mcache).refill(0x2027fa2000, 0x200000001e, 0x2027747078)
        go/src/runtime/mcache.go:122 +0x88
runtime.(*mcache).nextFree.func1()
        go/src/runtime/malloc.go:538 +0x28
runtime.systemstack(0x2027c35000)
        go/src/runtime/asm_arm64.s:241 +0x90
runtime.mstart()
        go/src/runtime/proc.go:1132

goroutine 5 [running]:
runtime.systemstack_switch()
        go/src/runtime/asm_arm64.s:190 +0x8 fp=0x4437c85460 sp=0x4437c85450
runtime.(*mcache).nextFree(0x2027fa2000, 0x2026fb361e, 0x2027c355e0, 0x2026ff0cc8, 0x2027c355e0)
        go/src/runtime/malloc.go:539 +0xa0 fp=0x4437c854b0 sp=0x4437c85460
runtime.mallocgc(0x380, 0x20276daec0, 0x20772c7301, 0x2026fc02c8)
        go/src/runtime/malloc.go:691 +0x738 fp=0x4437c85550 sp=0x4437c854b0
runtime.newarray(0x20276daec0, 0x4, 0x2027c35502)
        go/src/runtime/malloc.go:833 +0x60 fp=0x4437c85590 sp=0x4437c85550
runtime.makemap(0x2027689ca0, 0x10, 0x0, 0x0, 0x2029b474ab)
        go/src/runtime/hashmap.go:281 +0x27c fp=0x4437c855e0 sp=0x4437c85590

[ianlancetaylor]

CC @aclements @RLH

[tmm1]

mapSpans is the only place in the code-base where mheap.spans is allocated:

go/src/runtime/mheap.go

Lines 542 to 555 in 93322a5

	func (h *mheap) mapSpans(arena_used uintptr) {
	// Map spans array, PageSize at a time.
	n := arena_used
	n -= h.arena_start
	n = n / _PageSize * sys.PtrSize
	n = round(n, physPageSize)
	need := n / unsafe.Sizeof(h.spans[0])
	have := uintptr(len(h.spans))
	if have >= need {
	return
	}
	h.spans = h.spans[:need]
	sysMap(unsafe.Pointer(&h.spans[have]), (need-have)*unsafe.Sizeof(h.spans[0]), h.arena_reserved, &memstats.other_sys)
	}

I wonder if that means the mmap is failing for some reason on android?

[aclements]

I keep coming back to this issue and not making any progress. I just don't see how this can happen.

@tmm1, do you happen to get (or can you get) core dumps from any of these? I'd love to see the memory map at the time of the crash.

My initial hunch was that we needed a memory fence between mapping spans and updating h.arena_used in mheap.setArenaUsed. This would be a little crazy since I don't see how the mmap could proceed without a full memory fence of its own, but maybe. However, the panic in #22204 (comment) happens while the heap lock is held, which means it can't be racing with mheap.setArenaUsed.

I tried looking for patterns in the faulting addresses, but didn't turn up much. The heap layout should be:

[0x4000000000, 0x4020000000) spans
[0x4020000000, 0x4420000000) bitmap
[0x4420000000, 0xc420000000) heap

Which means the first three faults correspond to

0x4000017698 => heap byte  0x5da6000 (0x4425da6000)
0x4000006578 => heap byte  0x195e000 (0x442195e000)
0x400004bea8 => heap byte 0x12faa000 (0x4432faa000)

None of these are on any obvious boundaries.

The good news is Go 1.11 almost certainly can't have this problem because I've completely rewritten all of the code involved (https://golang.org/cl/85887).

[gopherbot]

Change https://golang.org/cl/85887 mentions this issue: runtime: use sparse mappings for the heap