[DML] DmlGraphFusionTransformer deletes the initializers, and therefore breaks SetOptimizedModelFilePath()

[sclavel]

Describe the issue

in /core/providers/dml/DmlExecutionProvider/src/DmlGraphFusionTransformer.cpp:64, DmlGraphFusionTransformer::ApplyImpl() removes the initializers from the graph, without checking if the graph is going to be saved later.

(usually the initializers are removed in SessionState::FinalizeSessionState(..., bool remove_initializers, ..), where remove_initializers is set to !saving_model)

This obviously completely breaks SetOptimizedModelFilePath().

I am not sure what is the easiest way to fix, maybe InferenceSession::TransformGraph() should have a bool can_remove_initializers parameter that's propagated down the line?

To reproduce

OrtSessionOptions* options = nullptr;
ort->CreateSessionOptions(&options);
ort->SetOptimizedModelFilePath(options, L"optimized.onnx");
ort->CreateSession(env, L"model.onnx", options, &model);

-> correctly create a file optimized.onnx, that can be used

OrtSessionOptions* options = nullptr;
ort->CreateSessionOptions(&options);
OrtSessionOptionsAppendExecutionProvider_DML(options, 0);
ort->SetOptimizedModelFilePath(options, L"optimized.onnx");
ort->CreateSession(env, L"model.onnx", options, &model);

-> creates a file optimized.onnx that doesn't have initializers and cannot be used

Expected: to be able to save an optimized file when using DirectML

Urgency

This issue has been present since at least 1.12, but it's completely breaking an important feature, so probably should be fixed ASAP

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.13.1

ONNX Runtime API

C

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

DirectML 1.9.1

[fdwr]

@sumitsays I had a related pending PR #11346 that avoided graph fusion when SetOptimizedModelFilePath had been called, but I held off knowing your transformer work would dramatically change how fusion occurred anyway. I don't expect that PR addresses this specific aspect of initializers, but you'll need parts of that PR to enable the model file to be reloadable (otherwise the .onnx file has just a large fused blob rather than individual ONNX operators).

Related issue: #8440

[sumitsays]

@sclavel Thank you for raising this issue. This is not a regression in the latest release or main branch, because DML EP never worked with reloading of optimized onnx file. I would also like to mention DML EP never worked with .ort format also. And the fact that, .ort format is neither EP agonistic nor GPU agonistic nor ORT version agonistic, an .ort file exported from any other EP or GPU, won't work on DML EP.
We are working on to support both optimized onnx file and .ort file format. Stay tuned!

[sclavel]

I understand saving the fused DML node might not be possible without some deep changes.

The ideal then would be to save the graph before DML fusion is applied, and then keep applying the fusion (and that wouldn't break existing provider-agnostic code paths that disable optimizations when loading an optimized graph, since DML fusion is mandatory and bypass the optimization settings).

Short of this, a short-term fix would be to at least throw an error in FinalizeSession when saveModel is set, and dml fusion has been applied (same as it throw a warning for nchwc now).

[sumitsays]

Hi @sclavel ,
We have fixed this issue as part of this #13913. From now DirectML EP supports export of optimized model. Additionally, user will also be able to disable the DMLGraphFusionTransformer explicitly by setting the optimization level <= L2.
The fix will be out in ORT1.14 by early February. In the meantime, you can build ORT from source and use this new feature.

Thank you.