runtime: occasional hard lockup / 100% CPU usage in go applications

[ssokol]

What version of Go are you using (go version)?

go version go1.19.1 linux/arm

Does this issue reproduce with the latest release?

Unknown. I've been having this issue since I started working on this project roughly 24 months ago. Same behavior on all go versions.

What operating system and processor architecture are you using (go env)?

Raspberry Pi OS v10.11 on armv7l

go env Output

GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go_path/pkg/mod"
GOOS="linux"
GOPATH="/root/go_path"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/go/pkg/tool/linux_arm"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2944008957=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm using go to build a set of services on the Raspberry Pi platform (Pi 4, CM4). Several of the higher duty-cycle services occasionally "run away", consuming upwards of 400% (100% x 4 cores) of available processing power. This usually happens after several hours or even a few days of running. When this happens, all of the goroutines in the application stop running, remaining in "runtime.gopark". (See the stack trace below.)

The program most given to this behavior is a network broadcaster. It subscribes to and receives data from a set of Redis pubsub channels and forwards that data to clients over UDP. The typical throughput is relatively low - only about 34 KB/second. It never gets above 38 KB/sec. It is run at a higher priority level, but it is not run as "realtime".

I've added a watchdog goroutine to try to detect the runway condition and terminate the service, but the watchdog timer never fires.

I tried pulling a profile using pprof but when the problem happens the http listener fails to respond.

I've tried tracing with strace but it doesn't return any data.

I've monitored open files with lsof and it does not appear to be leaking handles - open file count remains constant at 369.

I kicked it around with the Redigo team, and from what we can tell it's not something going on in Redigo - all the pubsub listeners simply get stuck in their waiting state.

What did you expect to see?

The application runs at its normal modest CPU load (5% - 8%)

What did you see instead?

The application sucks down as much CPU power as possible (395%)

Stack Trace

Too long to embed in this issue. Here it is as a gist:

https://gist.github.com/ssokol/b168de8b4546efd9b43a9d6af8538de9

[ianlancetaylor]

CC @golang/runtime

[prattmic]

Could you collect a core dump (ulimit -c unlimited and set GOTRACEBACK=crash)?

Then in gdb running thread apply all bt should show where the threads are stuck.

[ssokol]

Will do. Just to make sure I'm doing this correctly:

1. I've updated the systemd service file to export the GOTRACEBACK=crash var
2. I've updated the systemd service file to set the ulimit to unlimited

I tested out attaching to it from gdb (see below) and the output from the backtrace is minimal and not very useful as far as I can tell. Is there something else I need to do, either in my application or from gdb, to make the backtrace more verbose / useful?

Here's what I got. Note that this is in the normal state, not the runaway state.

root@pi4cm1(rw):/lib/systemd/system# gdb -p 12519
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 12519
[New LWP 12520]
[New LWP 12521]
[New LWP 12522]
[New LWP 12523]
[New LWP 12524]
[New LWP 12525]
[New LWP 12534]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
0x0007f2ec in ?? ()
(gdb) thread apply all bt

Thread 8 (Thread 0xa30ff440 (LWP 12534)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xa3afe440 (LWP 12525)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 0xa42ff440 (LWP 12524)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 0xa4efe440 (LWP 12523)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0xa56ff440 (LWP 12522)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0xa60ff440 (LWP 12521)):
#0  0x0007f5a4 in ?? ()
#1  0x0004727c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0xa6ab9440 (LWP 12520)):
#0  0x0007f2f0 in ?? ()
#1  0x000476d8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xb6f28080 (LWP 12519)):
#0  0x0007f2ec in ?? ()
#1  0x000475b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Apologies if this is a basic question, but I'm not at all familiar with using gdb on a go app.

[prattmic]

Will do. Just to make sure I'm doing this correctly:

1. I've updated the systemd service file to export the GOTRACEBACK=crash var
2. I've updated the systemd service file to set the ulimit to unlimited

That sounds right. Now when you send a SIGABRT it should generate a core file (though I'm not sure where systemd will put it). That said, if you can simply attach the PID with gdb (as you did below) when it is stuck that is fine too.

I tested out attaching to it from gdb (see below) and the output from the backtrace is minimal and not very useful as far as I can tell. Is there something else I need to do, either in my application or from gdb, to make the backtrace more verbose / useful?

This is odd, it should show the symbols there. This isn't a stripped binary, is it (e.g., go build -ldflags="-s -w")?

You can at least double check that these addresses are in the range of the binary (see info proc mappings), and perhaps find the addresses in objdump -d <binary>.

[ssokol]

Ah, it was a stripped binary. Rebuilt and running now with all the debugging symbols.

Here's the output from another test (again in the nominal not runaway state). Looks much better:

root@pi4cm1(rw):~/flightview/fvUnisocket# gdb -p 20193
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 20193
[New LWP 20194]
[New LWP 20195]
[New LWP 20196]
[New LWP 20197]
[New LWP 20198]
[New LWP 20199]
[New LWP 20206]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
466		SWI	$0
Loading Go Runtime support.
(gdb) thread apply all bt

Thread 8 (Thread 0xa2eff440 (LWP 20206)):
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x2102790, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x2102790) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052f04 in runtime.mPark () at /root/go/src/runtime/proc.go:2239
#4  runtime.stopm () at /root/go/src/runtime/proc.go:2239
#5  0x00054704 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2866
#6  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#7  0x000564bc in runtime.park_m (gp=0x2002c30) at /root/go/src/runtime/proc.go:3355
#8  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#9  0x00000004 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xa38ff440 (LWP 20199)):
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x2102550, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x2102550) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052f04 in runtime.mPark () at /root/go/src/runtime/proc.go:2239
#4  runtime.stopm () at /root/go/src/runtime/proc.go:2239
#5  0x00054704 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2866
#6  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#7  0x000564bc in runtime.park_m (gp=0x21043c0) at /root/go/src/runtime/proc.go:3355
#8  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#9  0x003e1690 in runtime.semtable ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 0xa42ff440 (LWP 20198)):
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x3ee154 <runtime.newmHandoff+12>, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x3ee154 <runtime.newmHandoff+12>) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052dc0 in runtime.templateThread () at /root/go/src/runtime/proc.go:2217
#4  0x00051648 in runtime.mstart1 () at /root/go/src/runtime/proc.go:1425
#5  0x000515b0 in runtime.mstart0 () at /root/go/src/runtime/proc.go:1382
#6  0x0007c56c in runtime.mstart () at /root/go/src/runtime/asm_arm.s:210

Thread 5 (Thread 0xa4cff440 (LWP 20197)):
#0  runtime.epollwait () at /root/go/src/runtime/sys_linux_arm.s:699
#1  0x0004727c in runtime.netpoll (delay=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /root/go/src/runtime/netpoll_epoll.go:126
#2  0x000545ec in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2829
#3  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#4  0x000564bc in runtime.park_m (gp=0x21043c0) at /root/go/src/runtime/proc.go:3355
#5  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#6  0x00000004 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0xa56ff440 (LWP 20196)):
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x202c790, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x202c790) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052f04 in runtime.mPark () at /root/go/src/runtime/proc.go:2239
#4  runtime.stopm () at /root/go/src/runtime/proc.go:2239
#5  0x00054704 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2866
#6  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#7  0x00056df4 in runtime.goexit0 (gp=0x2105e00) at /root/go/src/runtime/proc.go:3532
#8  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#9  0x00000004 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0xa60ff440 (LWP 20195)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x202c550, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x202c550) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052f04 in runtime.mPark () at /root/go/src/runtime/proc.go:2239
#4  runtime.stopm () at /root/go/src/runtime/proc.go:2239
#5  0x00054704 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2866
#6  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#7  0x000564bc in runtime.park_m (gp=0x2002c30) at /root/go/src/runtime/proc.go:3355
#8  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#9  0x00000004 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0xa6afa440 (LWP 20194)):
#0  runtime.usleep () at /root/go/src/runtime/sys_linux_arm.s:627
#1  0x0005aef8 in runtime.sysmon () at /root/go/src/runtime/proc.go:5145
#2  0x00051648 in runtime.mstart1 () at /root/go/src/runtime/proc.go:1425
#3  0x000515b0 in runtime.mstart0 () at /root/go/src/runtime/proc.go:1382
#4  0x0007c56c in runtime.mstart () at /root/go/src/runtime/asm_arm.s:210

Thread 1 (Thread 0xb6f69080 (LWP 20193)):
#0  runtime.futex () at /root/go/src/runtime/sys_linux_arm.s:466
#1  0x000475b8 in runtime.futexsleep (addr=0x3dc9b0 <runtime.m0+208>, val=0, ns=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/go/src/runtime/os_linux.go:69
#2  0x0001c220 in runtime.notesleep (n=0x3dc9b0 <runtime.m0+208>) at /root/go/src/runtime/lock_futex.go:160
#3  0x00052f04 in runtime.mPark () at /root/go/src/runtime/proc.go:2239
#4  runtime.stopm () at /root/go/src/runtime/proc.go:2239
#5  0x00054704 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /root/go/src/runtime/proc.go:2866
#6  0x00055cd8 in runtime.schedule () at /root/go/src/runtime/proc.go:3206
#7  0x000564bc in runtime.park_m (gp=0x2002c30) at /root/go/src/runtime/proc.go:3355
#8  0x0007c5cc in runtime.mcall () at /root/go/src/runtime/asm_arm.s:265
#9  0x00000004 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

[prattmic]

Yup, that looks correct!