Intermittent crash in ETW logging

[daveorourke]

Describe the issue

We're shipping onnxruntime.dll with our product, Camtasia. We originally shipped v1.18.0, which was stable. Later, we updated to v1.20.0, and we started getting crash reports from our telemetry.

To troubleshoot this issue, we built from the v1.20.0 tag so we could get .pdb debug symbols for onnxruntime.dll. We then reproduced the crash, which gave us the crash call stack shown here:

It's unclear (to us) what ETW event is triggering this crash, and we don't have reliable repro steps. But we do know the following:

1. The crash was introduced between v1.18.0 and v1.20.0 with the introduction of ETW logging.
2. The crash seems to occur at random (apparently from some ETW event), while our application is open, but can occur without any user interaction, and more often on resume from sleep.
3. The crash .dmp always has the same call stack as shown above.

We did note that the crash mentioned in this commit has a very similar call stack. Unfortunately, that commit was prior to the v1.20.0 tag and the crash persists.

Crashes by OSVersion

To reproduce

1. Launch Camtasia Windows 2025.1.1 (current version, as of 5/15/2025), which uses onnxruntime v1.20.0
2. Wait. 😐

As shown above, the crash occurs in onnxruntime.dll stemming from an ETW callback. But we don't have reliable repro steps to trigger it. ☹

Urgency

As of today (5/15/2025), we have over 2,000 crash reports from users who have voluntarily submitted them (users get a crash dialog with Send / Don't Send buttons). We think most users click "Don't Send", so the actual number is much higher.

We would prefer to use the Microsoft pre-built binaries, but we're unsure how to mitigate this issue. We could fork the code and comment out the ETW logging, and build onnxruntime.dll ourselves, but that's not ideal. We have tried to roll forward to v1.21.0, but ran into some other crash issues with our CI tests. We've considered rolling back to v1.18.0, but any security issue would force us to roll forward, so that workaround is temporary at best. We're open to any workarounds, if you have them.

Platform

Windows

OS Version

Windows 10 or 11. Most common on Win 11 24H2 (v10.0.26100). See above screenshot.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

https://github.com/microsoft/onnxruntime/tree/v1.20.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

DirectML.dll v1.15.2.0

[yuslepukhin]

Looks like the Etw call back is set before the session is initialized and the session logger is set.

Would it be possible to stacks from other threads? Thx!

[daveorourke]

Would it be possible to stacks from other threads? Thx!

@yuslepukhin Sure. Would a mini .dmp file work? I just got this same crash on my system, though I still don't know why. Our app was open, but idle at the time of the crash.

crash.dmp

[yuslepukhin]

Is this dump from the official release or from your binaries? Symbols for the official binaries are available on MS symbol servers.

[yuslepukhin]

Cc: @ivberg Looking at #21086 and it looks like it is not fixed.

[daveorourke]

@yuslepukhin That crash.dmp was generated by me, on my system, running an internal build of Camtasia, using the official release version of onnxruntime v1.20.24.1030.

The version of onnxruntime.dll that we're shipping came from https://github.com/microsoft/onnxruntime/releases/tag/v1.20.0

We're specificially using the DirectML version, pictured here:

Here's the file properties:

Symbols for the official binaries are available on MS symbol servers.

That's odd because I am using the MS symbol servers, yet the symbols don't resolve. Here's my symbol settings in VS:

For the screenshot in the description above, we had built onnxruntime.dll ourselves, from the v1.20.0 tag in order to get .pdb symbols, to see the top of the call stack. This was only necessary because we couldn't get the symbols to resolve from the MSFT symbol server.

[yuslepukhin]

I see. Unfortunately, even though they’re built from the same source, the symbols don’t match.
Could you please grab a dump from the release binaries when you get a chance? Thanks!

[daveorourke]

@yuslepukhin To be clear, we are using the release binaries for the DirectML version of onnxruntime, as I described above.

We noted that the DirectML v1.20 version does not include *.pdb debug symbols as some of the other posted binaries do. You can verify this yourself by downloading and unzipping the Microsoft.ML.OnnxRuntime.DirectML.1.20.0.nupkg from the v1.20 release page. There are no .pdb symbols in that .nupkg. By contrast, the onnxruntime-win-x64-1.20.0.zip does have .pdb symbols.

In case it helps, here's another crash2.dmp. This crash also happened on my machine, but while running Camtasia 25.1.1, our publicly available release build, which was using the same DirectML.1.20.0 version of onnxruntime.dll.

[yuslepukhin]

Let me produce a fix, then I can have a branch off 1.20 for you to try if you do not mind.

[yuslepukhin]

Please, try this branch.
https://github.com/microsoft/onnxruntime/tree/yuslepukhin/etw_crash

I this works, I will PR with details to main.

[daveorourke]

@yuslepukhin Thanks. We'll give it a try.

[daveorourke]

@yuslepukhin Good news... your branch seems to be working.

Here's what I did:

I pulled down your branch and from a VS Command Prompt, I built onnxruntime.dll and DirectML.dll, per the instructions here. This was the command line I used:

build.bat --config RelWithDebInfo --build_shared_lib --parallel --use_dml

I took the resulting onnxruntime.dll and DirectML.dll from this build, and used it with a build of Camtasia 25.1.1 that was previously crashing (with onnxruntime.dll v1.20), as described above. Since we didn't have repro steps, I left it running for 1.5 days and did not see any crashes.

So based on these results, it does seem to address the issue. Thank you for looking at it. We do appreciate it.