Crash with certain models in C++ DirectML, but not in Python DirectML

[divideconcept]

Describe the issue

Certain ONNX models work (load and inference) when using the DirectML provider in C++, but others don't, they either crash on load or on inference. However loading and infering the same models using the DirectML provider in Python works.

To reproduce

model1.onnx: Crash on Load

You can download model1.onnx from here.

In Python, this loads and infere fine with DirectML:

import onnxruntime as ort
import numpy as np
session=ort.InferenceSession("model1.onnx", providers=['DmlExecutionProvider'])
input=np.zeros((1,2,343980),dtype=np.float32)
outputs=session.run(None, {"input": input})
print(outputs[0].shape) #print (14, 2049, 340, 2)

However in C++, it crashes during loading:

#include "onnxruntime_cxx_api.h"
#include "dml_provider_factory.h"
Ort::SessionOptions sessionOptions;
OrtSessionOptionsAppendExecutionProvider_DML(sessionOptions, 0);
Ort::Env env;
Ort::Session session=Ort::Session(env, "model1.onnx", sessionOptions);

crash with the following message:

[E:onnxruntime:, inference_session.cc:1645 onnxruntime::InferenceSession::Initialize::<lambda_016a66eff4a24df6cdf369d701ac5914>::operator ()] Exception during initialization: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\AbiCustomRegistry.cpp(516)\onnxruntime.dll!00007FFB8ED2ED45: (caller: 00007FFB8ED53645) Exception(3) tid(17c4) 80070057 The parameter is incorrect.

model2.onnx: Crash on Inference

You can download model2.onnx from here.

In Python, this loads and infere fine with DirectML:

import onnxruntime as ort
import numpy as np
session=ort.InferenceSession("model2.onnx", providers=['DmlExecutionProvider'])
input=np.zeros((1,5600),dtype=np.float32)
outputs=session.run(None, {"input": input})
print(outputs[0].shape) #print (1, 5600, 1)

However in C++, it crashes during inference:

#include "onnxruntime_cxx_api.h"
#include "dml_provider_factory.h"
Ort::SessionOptions sessionOptions;
OrtSessionOptionsAppendExecutionProvider_DML(sessionOptions, 0);
Ort::Env env;
Ort::Session session=Ort::Session(env, "model2.onnx", sessionOptions);
Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);

const char* input_names[] = {"input"};
std::vector<int64_t> inputShape={1,5600};
std::vector<flaot> inputData(5600);
Ort::Value input_tensor = Ort::Value::CreateTensor<float>(memory_info, inputData, inputData.size(), inputShape.data(), inputShape.size());

const char* output_names[] = {"output"};
std::vector<int64_t> outputShape={1,5600,1};
std::vector<flaot> outputData(5600);
Ort::Value input_tensor = Ort::Value::CreateTensor<float>(memory_info, outputData, outputData.size(), outputShape.data(), outputShape.size());

Ort::RunOptions run_options;
stemsSession.Run(run_options, input_names, &input_tensor, 1, output_names, &output_tensor, 1);

crash with the following message:

[E:onnxruntime:, sequential_executor.cc:514 onnxruntime::ExecuteKernel] Non-zero status code returned while running Softmax node. Name:'/masknet/dual_mdl.0/intra_mdl/mdl/layers.0/self_att/att/Softmax' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2448)\onnxruntime.dll!00007FFB8ED1CB7A: (caller: 00007FFB8ED1C0CB) Exception(3) tid(4280) 80070057 The parameter is incorrect.

C++ Library Used

https://github.com/microsoft/onnxruntime/releases/download/v1.15.1/Microsoft.ML.OnnxRuntime.DirectML.1.15.1.zip

Urgency

Serious, as I can't offer hardware acceleration on Windows for my C++ software. CPU inference is much slower.

Platform

Windows

OS Version

11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.15.1

ONNX Runtime API

C++

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

No response

[divideconcept]

Ok, this seems very similar to #13429
So I eventually got it to work with C++, but to do so I had to grab the DirectML.dll that came with onnxruntime-directml on Python, located in python\Lib\site-packages\onnxruntime\capi\DirectML.dll and copy it next to onnxruntime.dll and my .exe.

This should probably require a proper rebundling of the release packages so that this dll is included along with onnruntime.dll. @wchao1115 @fdwr @yuslepukhin

[nums11]

Closing as resolved.