Skip to content

[Bug]: MIGraphX EP crashes in bundled amdhip64_7.dll (access violation, 0xc0000005) compiling a large model on RX 7900 XTX #15

Description

@Looong01

Which sample?

N/A — this is a Windows ML runtime crash in the MIGraphX execution provider itself (package MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8, EP version 1.8.57.0), not tied to any sample code in this repo. Filing here per the maintainers' guidance in case triage wants to redirect to microsoft/WindowsAppSDK; happy to move it there if preferred.

Steps to reproduce

  1. On a Windows machine whose only GPU is an AMD Radeon RX 7900 XTX (gfx1100), build a C++
    application using the ONNX Runtime C++ API (via Windows ML's EP catalog) that:
    • Discovers and registers the MIGraphX EP via WinMLEpCatalog (EnsureReady + GetLibraryPath),
      then Ort::Env::RegisterExecutionProviderLibrary("MIGraphX", ).
    • Selects the MIGraphX device via Ort::Env::GetEpDevices(), filtering for EP name
      "MIGraphX", and appends it with SessionOptions::AppendExecutionProvider_V2.
    • Before session creation, pins the ONNX graph's "batch" free dimension to a fixed value
      (e.g. 8) via SessionOptions::AddFreeDimensionOverrideByName("batch", 8), on a ~930MB
      ONNX graph (converted from a KataGo Go-playing neural network; the exact model isn't
      important, any moderately large graph with a pinned static batch dimension should do).
  2. Construct the Ort::Session (first-time compile, no on-disk compile cache present yet for
    this model/shape).
  3. Observe: CPU spikes to 100% for several minutes, host RAM climbs to several GB, VRAM usage
    stays at 0% throughout, then the process crashes with no exception and no error output.

This reproduced identically twice, from two separate process launches (once via an interactive
stdio/GTP-protocol subprocess, once via a one-shot batch-mode subcommand) — same crash, same
module, same offset both times.

Minimal repro code is not attached since this was found via an existing open-source project
(a fork of KataGo with a Windows ML backend); happy to extract a minimal standalone repro if
that would help triage — let me know.

Expected behavior

The model compiles successfully and inference proceeds, OR the failure surfaces as a catchable
ONNX Runtime exception (e.g. Ort::Exception / EP_FAIL) rather than a native access violation.

For reference, the identical model/code/EP-package-version compiles successfully (in ~9 minutes)
on a different machine with an integrated AMD GPU + NPU, so this appears specific to this
discrete-GPU (RX 7900 XTX) configuration.

Actual behavior

The process hard-crashes with an unhandled access violation. Windows Event Viewer (Application
log, Event ID 1000, source "Application Error") shows:

Faulting application name: katago.exe, version: 0.0.0.0, time stamp: 0x6a4fb5f2
Faulting module name: amdhip64_7.dll, version: 7.2.2606.20, time stamp: 0x69e9440d
Exception code: 0xc0000005
Fault offset: 0x0000000000465007
Faulting module path: C:\Program Files\WindowsApps
MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.57.0_x64__8wekyb3d8bbwe
ExecutionProvider\amdhip64_7.dll

Reproduced twice (different PIDs, different process launches) with the IDENTICAL faulting
offset both times — this rules out a random memory-pressure or race-condition explanation.
System RAM was 64GB with 55%+ free at time of testing, so this is not a simple OOM.

No exception is thrown into the calling application before the crash — the crash happens
entirely inside the EP's own bundled amdhip64_7.dll, downstream of the Ort::Session
constructor, so it cannot be caught by the caller's try/catch around session creation.

Windows version

Windows 11 25H2 OS Internal version: 26200.8737 ( Win32NT 10.0.26200.0 Microsoft Windows NT 10.0.26200.0)

Architecture

x64

Hardware (CPU / GPU / NPU)

Intel Core i7-9700, AMD Radeon RX 7900XTX

Execution provider

MIGraphX (explicitly selected via winmlProvider config option in the calling application; not using automatic EP-selection policy). EP name as reported by the catalog: MIGraphXExecutionProvider. Package: MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_8wekyb3d8bbwe, version 1.8.57.0.

Windows ML / Windows App SDK package version

Microsoft.Windows.AI.MachineLearning 2.0.297-preview

Diagnostic logs (optional)

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions