runtime: "fatal: morestack on g0" on Linux

[shenli]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.9.1 linux/amd64

Does this issue reproduce with the latest release?

I do not know how to reproduce it. But it occurs twice.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build898823429=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

Run a binary as daemon service.

What did you expect to see?

The process runs forever.

What did you see instead?

I get "fatal: morestack on g0" from stderr. The process is still there but does not respond anymore. When I use curl http://ip:port/debug/pprof/goroutine?debug=1 to check the stack, but it halts. There is nothing useful in stderr or dmesg.

I tried pstack and got the following result. It is very strange that there is only one thread alive. I have googled it but get nothing useful.

Thread 1 (process 12230):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
#1  0x00000000004293f2 in runtime.futexsleep (addr=0x1b0a950 <runtime.m0+272>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2  0x000000000040f9ab in runtime.notesleep (n=0x1b0a950 <runtime.m0+272>) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x00000000004315f5 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1680
#4  0x00000000004327d2 in runtime.findrunnable (gp=0x45cbd1 <runtime.goexit+1>, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2135
#5  0x000000000043328c in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2255
#6  0x00000000004335a6 in runtime.park_m (gp=0xca01842780) at /usr/local/go/src/runtime/proc.go:2318
#7  0x0000000000459ffb in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:286
#8  0x0000000001b0a100 in ?? ()
#9  0x00007ffdc5be26e0 in ?? ()
#10 0x0000000001b0a100 in ?? ()
#11 0x00007ffdc5be26d0 in ?? ()
#12 0x0000000000430544 in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1152
#13 0x0000000000459ec1 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:186
#14 0x000000000000000a in ?? ()
#15 0x00007ffdc5be2718 in ?? ()
#16 0x000000000000000a in ?? ()
#17 0x00007ffdc5be2718 in ?? ()
#18 0x0000000000000000 in ?? ()

[shenli]

My Linux kernel is "Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux"

go/src/runtime/os_linux.go

Line 45 in bf9ad70

futex(unsafe.Pointer(addr), _FUTEX_WAIT, val, nil, nil, 0)

	// Some Linux kernels have a bug where futex of
	// FUTEX_WAIT returns an internal error code
	// as an errno. Libpthread ignores the return value
	// here, and so can we: as it says a few lines up,
	// spurious wakeups are allowed.
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT, val, nil, nil, 0)
		return
	}

Could I say that I meet the bug of Linux kernel?

[cznic]

Could I say that I meet the bug of Linux kernel?

Hard to tell w/o a reproduction case.

[shenli]

@cznic Thanks! I am trying to reproduce it. But I am not sure whether I could make it.

[odeke-em]

I'll also page some other folks to be aware of this as we wait for more information
/cc @randall77 @aclements

[shenli]

I upgrade my gdb to 8.0 and find that there are 74 threads but all of them are waiting here: runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439

[davecheney]

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

[shenli]

@davecheney Thanks!
It occurs in the production environment. We could not enable race detection. I will try to build a testing environment to reproduce it.

[zhexuany]

We just figured out how to print goroutine using gdb. Below is partial stack of all goroutine:

---Type <return> to continue, or q <return> to quit---
  208832207 waiting  runtime.gopark
  208816967 waiting  runtime.gopark
  208824056 waiting  runtime.gopark
  208825674 waiting  runtime.gopark
  208813050 waiting  runtime.gopark
  208821762 waiting  runtime.gopark
  208845379 waiting  runtime.gopark
  208803855 waiting  runtime.gopark
  208819750 waiting  runtime.gopark
  208820173 waiting  runtime.gopark
  208830287 waiting  runtime.gopark
  208835489 waiting  runtime.gopark
  208818500 waiting  runtime.gopark
  208844090 waiting  runtime.gopark
  208803883 waiting  runtime.gopark
  208815260 waiting  runtime.gopark
  208843403 waiting  runtime.gopark
  208817095 waiting  runtime.gopark
  208819772 waiting  runtime.gopark
  208830692 waiting  runtime.gopark
  208808392 waiting  runtime.gopark
  208830801 waiting  runtime.gopark
  208843133 waiting  runtime.gopark
  208839727 waiting  runtime.gopark
  208826235 waiting  runtime.gopark
  208836382 waiting  runtime.gopark
  208828454 waiting  runtime.gopark
  208817075 waiting  runtime.gopark
  208829301 waiting  runtime.gopark

All goroutines are wating on gopark. In. addition, all os threads are waiting on futex.
We came up with two possible reasons: 1. deadlock, 2. g0 is crashed which implies there is no more goroutine can schedule other goroutines.

[zhexuany]

Deadlock could be caused from kernel bug because our binary in running against linux 3.10 which has a kernel bug causing potential deadlock. For second reason, we are confused why all thread's schedulers are crashed. Per my understanding, all threads have their own g0. The process is still running. Any suggestions about how to debug this will be really appreciated. Of course, tell us what is reason behind this will be also really appreciated.

[ianlancetaylor]

It's normal for an inactive goroutine to be sitting in gopark. It's normal for an inactive thread to be waiting on a futex.

Your original report, morestack on g0, is a clear sign of a bug or of memory corruption. A deadlock could simply be the result of a program bug.

You suggest that a g0 could crash, causing deadlock, but in general that can't happen. If a g0 crashes, the entire program crashes, as in your original report.

[aclements]

@shenli, it would be useful if you could dump all of the thread stacks from gdb using thread apply all bt.

Unfortunately, I wouldn't expect the info goroutines command to do much useful since the g0 isn't a regular user goroutine and hence isn't printed by info goroutines.