Add crash and server restart alert

[finestructure]

We should add an alert for server restarts

[ NOTICE ] Server starting on http://0.0.0.0:80 [component: server]

We've seen crashes today that led to swift-backtrace locking up the nodes for a while:

We don't have good visibility when restarts happen - it's a testament really to how stable the processes have been, we've never really had issues with crashes in the server process.

[finestructure]

I don't think we want swift-backtrace running by default in prod with those performance characteristics. On staging, yes, but in prod I'd want to enable it explicitly as a last resort to track down issues. We ended up shooting down nodes left, right, and centre due to it taking so long to wrap up.

[finestructure]

https://github.com/apple/swift/blob/main/docs/Backtracing.rst

SWIFT_BACKTRACE="enable=no" I think.

[finestructure]

We had another instance this morning where a node was completely locked up for hours, running swift-backtrace at 100% CPU:

What's also concerning is that we're not seeing any traces anymore in our logs even when backtrace doesn't lock up. We used to get unsymbolicated ones at least but now they're gone.

[finestructure]

Also look for Backtrace took as a crash alert.

[tayloraswift]

We had another instance this morning where a node was completely locked up for hours, running swift-backtrace at 100% CPU:

i have seen something sporadically similar to this ever since upgrading to swift 5.9. did you see swift-backtrace in top while this was occurring? as far as i understand, swift-backtrace should not take more than a minute to complete.

[finestructure]

Yes, exactly - like in the screenshot at the very top. My understanding is that the hang was perhaps due a misconfiguration. We're statically linking but hadn't bundled up and reference the backtrace binary. But it's possible that this happens regardless, it's hard to tell at this point.