Alternative backends on Windows don't catch crashes caused by `abort()`

[BenjiHansell]

Using Breakpad or inproc on Windows results in most crashes not being captured.

When does the problem happen

• During build
• During run-time
• When capturing a hard crash

Environment

• OS: Only Windows (in my case, Windows 10, 64-bit in VirtualBox)
• Compiler: MSVC 19
• CMake version and config: 3.21.2 -D SENTRY_BACKEND=breakpad -D BUILD_SHARED_LIBS=OFF

Steps To Reproduce

Just a simple boilerplate setup does it. Using abort() to cause the crash.

int main() {
  std::cout << "hello world" << std::endl;

  sentry_options_t* options = sentry_options_new();
  sentry_options_set_dsn(options, 'dsn redacted');
  sentry_options_set_debug(options, 1);

  sentry_init(options);

  std::this_thread::sleep_for(std::chrono::milliseconds(2000));

  abort();

  sentry_close();
}

Log output

This is the logs when it doesn't work. As you can see, no crash is captured.

hello world
[sentry] INFO using database path "C:\Users\vagrant\NonRepos\SentryNativeTutorial\.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG background worker thread started
The terminal process "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command ./install/bin/tutorial.exe" terminated with exit code: 1.

However, if I swap out abort(); for

  volatile char* invalid_memory = nullptr;
  *invalid_memory = '\0';  // crash

to invoke a different kind of crash, I get

hello world
[sentry] INFO using database path "C:\Users\vagrant\NonRepos\SentryNativeTutorial\.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG background worker thread started
[sentry] INFO entering breakpad minidump callback
[sentry] DEBUG merging scope into event
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] INFO crash has been captured
The terminal process "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command ./install/bin/tutorial.exe" terminated with exit code: 1.

That is the first run, and on subsequent runs I see the envelope is uploaded and appears in the Sentry UI.

Similarly, if I change over to use crashpad as the backend, things work fine (but crashpad is not an option for me on this project). Things also work fine on Linux and Mac with the breakpad backend.

In my project, there's a lot of assertions so most crashes are caused in the way that Windows can't handle them.

For now, my hacky workaround is to register my own signal handler like so:

[[noreturn]] void windows_signal_handler(int signal) {
  volatile char* invalid_memory = nullptr;
  *invalid_memory = '\0';  // raising SEH exception EXCEPTION_ACCESS_VIOLATION_WRITE
  // unreachable
}

std::signal(SIGABRT, windows_signal_handler);

This results in the full stack trace appearing on the UI, albeit with an extra frame or two.

[jan-auer]

Thanks for sharing this observation. When compared with Breakpad, is using the inproc backend an option for you with its lack of minidumps? I'm assuming that you can't use Crashpad because you need a single executable or even process.

We'll make sure to add this to the inproc backend and expand our test matrix. While we say that the inproc backend technically supports Windows, it's not on par with Crashpad. We can further clarify this in the documentation. (cc @Swatinem)

Since we're planning to remove the Breakpad backend eventually, we're not going to add this there. Of course, we

[BenjiHansell]

Thanks for sharing this observation. When compared with Breakpad, is using the inproc backend an option for you with its lack of minidumps? I'm assuming that you can't use Crashpad because you need a single executable or even process.

We'll make sure to add this to the inproc backend and expand our test matrix. While we say that the inproc backend technically supports Windows, it's not on par with Crashpad. We can further clarify this in the documentation. (cc @Swatinem)

Since we're planning to remove the Breakpad backend eventually, we're not going to add this there. Of course, we

Thanks for this clarification.

I don't think inproc is an option for us. We've gone with Breakpad due to the lack of minidumps from inproc. I hadn't realised the Breakpad backend was going to be removed though.

Is there perhaps any plan to add minidump functionality to the inproc backend before deprecating Breakpad?

P.S. The reason we can't use Crashpad is actually more nuanced. Yes the extra binary is inconvenient, but we can (and previously did) work with that. The bigger problem is that we are now using a custom transport, and Crashpad doesn't send crashes through Sentry Native's transport, instead it makes its own HTTP connection in the Crashpad process. HTTP isn't a reliable option in our high-security environment.

[jan-auer]

I don't think inproc is an option for us. We've gone with Breakpad due to the lack of minidumps from inproc.

Thanks for clarifying. I think we'll still try to capture abort() from the inproc backend since its a very reasonable fix, but then this clearly won't resolve your issue.

I hadn't realised the Breakpad backend was going to be removed though.

We haven't started advertising this yet since it's not on any immediate roadmap. However, it's our understanding that the Breakpad client is in maintenance mode even at Google, and it adds considerable overhead maintaining both. For now, you can rely on the Breakpad backend for a little. That said:

The bigger problem is that we are now using a custom transport, and Crashpad doesn't send crashes through Sentry Native's transport, instead it makes its own HTTP connection in the Crashpad process. HTTP isn't a reliable option in our high-security environment.

This is interesting insight, thanks for bringing this up. While this now strays from the original topic, I think there might be alternate solutions. Just so I understand, the issue with Crashpad's transport is purely that it uses HTTP instead of HTTPS? Phrased differently, if we fix that, would you be able to go with Crashpad?

[BenjiHansell]

This is interesting insight, thanks for bringing this up. While this now strays from the original topic, I think there might be alternate solutions. Just so I understand, the issue with Crashpad's transport is purely that it uses HTTP instead of HTTPS? Phrased differently, if we fix that, would you be able to go with Crashpad?

Actually it's not a HTTPS thing. To be more specific, we're using gRPC for our custom transport because that's pretty much the only protocol that is permitted to leave the secure area of our network. So, we have a custom transport which communicates with a gRPC->HTTP proxy which passes envelopes to the Sentry cloud.

[jan-auer]

Makes sense, thanks. Let me circle back with @Swatinem which of these options would make most sense, and it could be multiple:

• At very least, document limitations of secondary backends.
• Patch a signal handler for std::abort into Breakpad's win client in a minimal effort, double-checking past work. It's actually surprising to me that it's not there yet, but we'll have to take a deeper look.
• Same for the in-proc backend
• Add a way to provide a transport function to crashpad_handler. This comes with the more general challenge of customizing the handler process which we haven't tackled much, yet. I think the biggest challenge is around build tooling and a neat API for this.

Until then, you might have another workaround on your end. The SDK always vendors the getsentry branch of our Crashpad fork. The Windows HTTP transport, particularly ExecuteSynchronously is defined here. You could either provide your own implementation of the HttpTransport base class and swap that in, or patch out the Windows implementation to run gRPC internally. The file's history seems fairly stable, which might make a custom patch feasible.

[BenjiHansell]

Makes sense, thanks. Let me circle back with @Swatinem which of these options would make most sense, and it could be multiple:

• At very least, document limitations of secondary backends.
• Patch a signal handler for std::abort into Breakpad's win client in a minimal effort, double-checking past work. It's actually surprising to me that it's not there yet, but we'll have to take a deeper look.
• Same for the in-proc backend
• Add a way to provide a transport function to crashpad_handler. This comes with the more general challenge of customizing the handler process which we haven't tackled much, yet. I think the biggest challenge is around build tooling and a neat API for this.

Until then, you might have another workaround on your end. The SDK always vendors the getsentry branch of our Crashpad fork. The Windows HTTP transport, particularly ExecuteSynchronously is defined here. You could either provide your own implementation of the HttpTransport base class and swap that in, or patch out the Windows implementation to run gRPC internally. The file's history seems fairly stable, which might make a custom patch feasible.

Many thanks. For now we'll stick with Breakpad as we've got that implemented and working for the time being. We'll bear the Crashpad patch solution in mind for the future though.

[brianmichel]

Just wanted to chime in that we are working through an interesting integration (using Swift on Windows) and trying to port some bindings from the native client into a small Swift library and encountered the same exact issue. Is the recommended work around to just install a signal handler in client code and trigger a different exception?

We could use breakpad I suppose, but it sounds like that support is going away?