proposal: runtime: add a default, non-destructive signal handler that creates a stack dump

[kgibm]

Proposal Details

By default, Go produces a stack dump and exits on various signals:

A SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGSTKFLT, SIGEMT, or SIGSYS signal causes the program to exit with a stack dump.

It is commonly desired to create such a stack dump using a mechanism such as a signal but without exiting. Such thread dumps may be used to investigate hangs or as a rudimentary sampling profiler. Such built-in handlers are common in other languages such as Java (e.g. using SIGQUIT).

One way for a program to handle this requirement is to handle a signal such as SIGUSR1, generate a stack dump by calling pprof.Lookup on the goroutine profile and then write that to a file with Profile.WriteTo.

Alternatively, native debuggers and tools such as pstack may be used but they generally lack the detail of Go stack dumps.

This proposal is for Go to have a built-in signal handler that non-destructively writes a thread dump to stderr in response to a signal such as SIGUSR2. This would be a behavior change that may cause unintended side effects for any programs that already have a SIGUSR2 handler.

[bcmills]

In theory you could build this as a library using runtime.Stack and signal.Notify.

[mauri870]

I also think this is fairly simple to implement in a separate package, and one might depend on different conditions for it to trigger. Here we are discussing signal handlers, but maybe someone else wants to create a dump when memory reaches a certain threshold, or some other condition is reached. I like the idea of a general api involving runtime.Stack that is triggered by some custom condition and writes to a io.Writer of choice.

[ianlancetaylor]

I agree that this seems easily implementable outside of the standard library. It also doesn't seem like a common requirement. Is there a good reason to put this in the standard library? See https://go.dev/doc/faq#x_in_std.

[kgibm]

It also doesn't seem like a common requirement.

In my experience, enterprise software is exponentially more difficult to support and maintain if simple diagnostics need to be bolted on rather than provided "in the box" (I'm not a Go expert, but I presume "in the box" is equivalent to "inside the standard library"). This feature request was driven by a production incident where we could not get the simple diagnostic of a thread dump that shows what all threads are doing (the program was in a container, so we couldn't easily try pstack either, though I don't know if that even works well with Go). I agree that this feature is easily implementable outside the shared library and that's what we did (though it would be nice if it was provided as a separate package as mentioned above); however, it took weeks to roll it out into production which is the main issue with such bolt-ons. Such built-in handlers are common in other languages such as Java (e.g. using SIGQUIT).