[Performance] Override free dimension not work in onnxruntime_perf_test if optimization level is set to 0 (disabled)

[qwu16]

Describe the issue

In onnxruntime_perf_test tool, -f is used to override free dimensions for dynamic input shape, it works well when optimization level is set to > 0. When optimization level is set to 0 and running onnxruntime_perf_test.exe with -f parameter to collect performance data, there is a hint: Overriding dimension with name, it seems that the free dimension is set successfully, but in fact it will not work.
In GenerateTransformers function of graph_transformer_utils.cc, FreeDimensionOverrideTransformer will only be added in optimization level > 0. So the message Overriding dimension with name is confusing and there is no way to override the free dimensions only if optimization level is set to 0. In some performance data collecting, we only need to fix the free dimensions.

To reproduce

Use onnxruntime_perf_test.exe -I -r 10 -o 0 -f batch_size:2 -f sequence:77 sd-v1.5-text-encoder.onnx to test the dynamic shape model.

Urgency

No response

Platform

Windows

OS Version

Windows1 11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.22.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

No

[qwu16]

This issue can be reproduced with models of any dynamic shape input and optimization level set to 0 when testing with onnxruntime_perf_test.exe.

[skottmckay]

You need at least the 'basic' (level 1) transformers to run for the override to be applied.

Why is it a problem to run those? It happens during model load and won't affect perf data. Level 1 optimizers only use ONNX operators.

[huningxin]

Using level 0 is because some EPs, like OpenVINO EP, recommend to disable ORT optimizations: https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html#onnxruntime-graph-level-optimization

In most cases it has been observed that passing the ONNX input graph as it is without explicit optimizations would lead to best possible optimizations at kernel level by OpenVINO™. For this reason, it is advised to turn off high level optimizations performed by ONNX Runtime for OpenVINO™ Execution Provider.

It looks like overriding free dimensions conflicts with this requirement.

But if onnx runtime perf test tool can display a warning when using -f and -o 0 at the same time, that would be useful.

[skottmckay]

This could maybe be used from onnxruntime_perf_test.

onnxruntime/onnxruntime/core/session/inference_session.h

Lines 253 to 263 in 849eee8

	/**
	* Filter the enabled optimizers (either transformer or rewrite rule) using optimizers_to_disable.
	* For an optimizer to be enabled, it must be allowed at the current optimization level (as specified in
	* session options), and NOT in optimizers_to_disable.
	* This allows finer grained control of the enabled/disabled optimizations.
	* Must be called before Initialize() to take effect.
	*
	* Calling this API is optional.
	* @return OK if success.
	*/
	common::Status FilterEnabledOptimizers(InlinedHashSet<std::string>&& optimizers_to_disable);

It has previously been used to disable specific optimizers, but could potentially be augmented to take a bool to invert the behavior (if set, only enable the optimizers in the list).

If onnxruntime_perf_test had a new argument to specify optimizers to enable/disable, and you limited the optimization level to 0, it should do what you need.

Alternatively, another option would be to fix the dimensions in the model using https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html

[qwu16]

This could maybe be used from onnxruntime_perf_test.

onnxruntime/onnxruntime/core/session/inference_session.h

Lines 253 to 263 in 849eee8

/**
* Filter the enabled optimizers (either transformer or rewrite rule) using optimizers_to_disable.
* For an optimizer to be enabled, it must be allowed at the current optimization level (as specified in
* session options), and NOT in optimizers_to_disable.
* This allows finer grained control of the enabled/disabled optimizations.
* Must be called before Initialize() to take effect.
*
* Calling this API is optional.
* @return OK if success.
*/
common::Status FilterEnabledOptimizers(InlinedHashSetstd::string&& optimizers_to_disable);
It has previously been used to disable specific optimizers, but could potentially be augmented to take a bool to invert the behavior (if set, only enable the optimizers in the list).

If onnxruntime_perf_test had a new argument to specify optimizers to enable/disable, and you limited the optimization level to 0, it should do what you need.

Alternatively, another option would be to fix the dimensions in the model using https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html

Thanks for the suggestion. We tried to use https://onnxruntime.ai/docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.html to fix the dimensions, but it seems error will happen during converting models like phi3, merged models and models whose input shape dynamic value is defined like "sequence + 1". After evaluation, we decided to use optimization level 1 to do the testing.