Reporting Unrecovered Panics

[chodges15]

Problem Statement

The current SDK will not report a panic if it isn't caught in a recover. The following example does not report any errors to Sentry. Panics and unhandled crashes are arguably the highest value errors an SRE would want to monitor and alert on in a production environment.

By default, Go will print the panic trace to stderr before exiting, but there is no opportunity for another go routine to recover the panic before the process terminates (note the absence of "Main exiting" in the output below).

There are a whole host of reasons a go program could panic, from data races to nil pointer dereferences, and there does not seem to be a good way to enforce panic recovery in every single go routine a developer might add to the codebase.

package main

import (
	"log"
	"time"
	"github.com/getsentry/sentry-go"
)

func main() {
	// Initialize Sentry SDK
	err := sentry.Init(sentry.ClientOptions{
		Dsn: "https://your-dsn@sentry.io/your-project",
		Debug: true,
	})
	if err != nil {
		log.Fatalf("sentry.Init: %s", err)
	}
	defer sentry.Flush(2 * time.Second)

	// Main goroutine recovery attempt (won't help)
	defer func() {
		if r := recover(); r != nil {
			log.Println("Main recovered:", r)
		}
	}()

	// Start problematic goroutine
	go func() {
		panic("goroutine panic - unrecovered")
	}()

	// Keep main alive to observe behavior
	time.Sleep(1 * time.Second)
	log.Println("Main exiting")
}

Ouputs:

...
[Sentry] 2025/02/20 16:41:01 Integration installed: GlobalTags
panic: goroutine panic - unrecovered

goroutine 24 [running]:
main.main.func2()
	/Users/chodges/go-panic/panic.go:29 +0x2c
created by main.main in goroutine 1
	/Users/chodges/go-panic/panic.go:28 +0xbc

Solution Brainstorm

Triggering a core dump is one way to get around this problem and report the error to Sentry, however that has its own set of drawbacks:

• Exposes sensitive data
• Generates large files which could fill the disk
• High file I/O could compromise performance of the compute node

Another option would be to take a config option in the SDK initialization that has a file path to stderr output for the process. Then on SDK startup, it could go through the stderr output and look for panic stack traces and report them to Sentry. Parsing logs for errors is messy, but given how Golang handles unrecovered panics there is really nothing that can be done with error handling post-panic and before process termination.

[cleptric]

Our current thinking to support this would be some sort of wrapper program.
./sentry-panic-monitor <your-program> <your-program-args>, which would then parse and report panics to Sentry.
We do have some internal prototype which we still need to flesh out more, but would this be something that would work for you?

[chodges15]

Hey @cleptric - yes, that makes a lot of sense and would work for us. When it's ready we'd like to build it from source, perhaps this wrapper would be included in this SDK repo?

[cleptric]

Yes, we'll add it to this repository.