cmd: Log goroutines (stack traces) on SIGUSR1

[mitjat]

Resolves https://app.clickup.com/t/866ahha15

This PR maxes the nexus binary respond to the SIGUSR1 os signal by logging the stack traces of all goroutines.

A good way to read the stack traces is to pass the (last minute of) logs through on of:

| jq -r  '.goroutines_all | select(.)'
| jq -r  '.goroutines_block | select(.)'
| jq -r  '.goroutines_mutex | select(.)'

Motivation:
Currently, the only way to obtain the stacks of goroutines is to send a SIGQUIT, which also terminates the program. And in a way that does not safely close the KVStore , at that. When analyzers get inexplicably stuck (see e.g. https://app.clickup.com/t/866ahh8xy), the list of goroutines can be very helpful for diagnosing the live system.

Tested locally.

[pro-wh]

do we need to tell the go runtime that we don't have to wait for this goroutine to exit before exiting?

[mitjat]

do we need to tell the go runtime that we don't have to wait for this goroutine to exit before exiting?

I don't think so. It hasn't come up in my testing. I believe Go doesn't have an implicit all_goroutines.Wait() appended to the end of main(). See also this mini demo; it prints "Program exited" immediately.

[pro-wh]

ok