runtime: does not traceback goroutine in systemstack

[dvyukov]

Run the following program:

package main

func main() {
    for {
        _ = make([]byte, 1<<20)
    }
}

and kill it with SIGABRT. Most of the it crashes as follows. I.e. stack of the main goroutine is not unwound.

SIGABRT: abort
PC=0x447b71 m=0

goroutine 0 [idle]:
runtime.futex(0x4af628, 0x0, 0x0, 0x0, 0x0, 0x4af140, 0x0, 0x0, 0x4077dd, 0x4af628, ...)
    ?:0 +0x21 fp=0x7fff603b11b8 sp=0x7fff603b11b0
runtime.futexsleep(0x4af628, 0x0, 0xffffffffffffffff)
    src/runtime/os1_linux.go:40 +0x53 fp=0x7fff603b1200 sp=0x7fff603b11b8
runtime.notesleep(0x4af628)
    src/runtime/lock_futex.go:142 +0xad fp=0x7fff603b1228 sp=0x7fff603b1200
runtime.stopm()
    src/runtime/proc1.go:1059 +0x127 fp=0x7fff603b1250 sp=0x7fff603b1228
runtime.exitsyscall0(0xc208000a00)
    src/runtime/proc1.go:2017 +0x16e fp=0x7fff603b1278 sp=0x7fff603b1250
runtime.mcall(0x7fff603b12a0)
    src/runtime/asm_amd64.s:204 +0x5e fp=0x7fff603b1288 sp=0x7fff603b1278

goroutine 1 [running]:
    goroutine running on other thread; stack unavailable

The main goroutine is actually at this point:

goroutine 1 [running]:
runtime.systemstack_switch()
    src/runtime/asm_amd64.s:216 fp=0xc20802dde0 sp=0xc20802ddd8
runtime.mallocgc(0x100000, 0x4516a0, 0x1, 0xc209254000)
    src/runtime/malloc.go:632 +0x8d9 fp=0xc20802deb0 sp=0xc20802dde0
runtime.newarray(0x4516a0, 0x100000, 0xc209354000)
    src/runtime/malloc.go:742 +0xce fp=0xc20802def0 sp=0xc20802deb0
runtime.makeslice(0x44ea00, 0x100000, 0x100000, 0x0, 0x0, 0x0)
    src/runtime/slice.go:32 +0x16c fp=0xc20802df40 sp=0xc20802def0
main.main()
    crash.go:5 +0x3c fp=0xc20802df90 sp=0xc20802df40

This makes it very hard to debug hanging programs. The stack must be unwound.

We leave g in running status during systemstack. This is probably part of the problem. Another part is why the goroutine does not return from systemstack during freezetheworld.

[dvyukov]

/cc @rsc @RLH @aclements

[randall77]

freezetheworld can't force a return from the system stack. The signal handler runs on the same M as the user code, so the user code can't make any forward progress.

A simple fix to this problem is to print the stack if the G's M is identical to the caller's M. The G can't be running, and crucially modifying its stack, if that is the case. (Unless of course the two Gs are identical, but we already skip printing in that case.)

[gopherbot]

CL https://golang.org/cl/9435 mentions this issue.

[dvyukov]

I still got "goroutine running on other thread; stack unavailable" 2 times out of 6.

[gopherbot]

CL https://golang.org/cl/9551 mentions this issue.