[Bug] Inconsistent Results After ONNX Runtime Optimization

[Thrsu]

Describe the issue

I am encountering an issue where the output of the model after optimization using ONNX Runtime is inconsistent with the original model. Specifically, the optimization process leads to mismatched results for one of the outputs, v5_0, while others remain consistent.

• Actual Behavior:

AssertionError: 
Not equal to tolerance rtol=0.001, atol=0.001

Mismatched elements: 18 / 918 (1.96%)
Max absolute difference: 867669248
Max relative difference: inf
 x: array([[-867669248,      32714, -867669248,      32714, -867669248,
             32714,          0,          0,          0,          0,
                 0,          0,          0,          0,          0,...
 y: array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0],...

• Expected Behavior:
  The optimized model should produce identical results for all outputs when compared to the original model, within the specified tolerance.

To reproduce

1. Download the model
2. run the following script:

import onnx
import onnxruntime as ort
import numpy as np
from onnxruntime.transformers import optimizer

model_path = "inconsis1.onnx"
optimized_model_path = f"./opt.onnx"
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
this_provider_list = ort.get_available_providers()

original_session = ort.InferenceSession(model_path, sess_options, providers=this_provider_list)
input_data = {"v2_0": np.random.rand(1, 1).astype(np.int32), "v9_0": np.random.rand(1, 6, 51, 1).astype(np.int32)}
output_names = [output.name for output in original_session.get_outputs()]
original_result = original_session.run(output_names, input_data)

optimized_model = optimizer.optimize_model(model_path, opt_level=99, use_gpu=True)
optimized_model.save_model_to_file(optimized_model_path)
optimized_session = ort.InferenceSession(optimized_model_path, providers=this_provider_list)
optimized_model = onnx.load(optimized_model_path)
optimized_result = optimized_session.run(output_names, input_data)

for r1, r2 in zip(original_result, optimized_result):
    np.testing.assert_allclose(r1, r2, atol=1e-3, rtol=1e-3)

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

5c1b7cc

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[skottmckay]

@tianleiwu would you be able to take a look?

Is there a reason you need to use the external python optimization script vs. letting the runtime optimizations run on the model?

[FFchopon]

@tianleiwu would you be able to take a look?

Is there a reason you need to use the external python optimization script vs. letting the runtime optimizations run on the model?

Really sorry to bother, I'm thinking that using the external Python optimization script could help have finer control over the model modifications and better isolate the specific components that may be causing discrepancies.

[FFchopon]

I encountered similar problem. I reproduced the script and found that the inconsistency occurred most of the time, but not every time. I'm very confused. Could anyone help me?
@yuslepukhin @snnn

[microsoft-github-policy-service]

Applying stale label due to no activity in 30 days