different results between the CPUExecutionProvider and CUDAExecutionProvider when onnxruntime executes a valid model

[coffezhou]

Describe the issue

For the following onnx model,

the results of onnxruntime with the CPUExecutionProvider are as follows:

ONNXRuntime:
 [array([[[[0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         ...,
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256]],

        [[0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         ...,
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256],
         [0.15210256, 0.15210256, 0.15210256, ..., 0.15210256,
          0.15210256, 0.15210256]],

        [[0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494],
         [0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494],
         [0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494],
         ...,
         [0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494],
         [0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494],
         [0.69579494, 0.69579494, 0.69579494, ..., 0.69579494,
          0.69579494, 0.69579494]]]],
      shape=(1, 3, 224, 224), dtype=float32)]

However, I run onnxruntime with the CUDAExecutionProvider, the results are all nan:

ONNXRuntime:
 [array([[[[nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         ...,
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan]],

        [[nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         ...,
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan]],

        [[nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         ...,
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan]]]],
      shape=(1, 3, 224, 224), dtype=float32)]

I am not sure that this issue is due to the difference between the CPU and CUDA.

To reproduce

Environment

OS: Ubuntu 20.04
onnxruntime: 1.23.0.dev20250515001
CUDA: cuda-12.2.2::cuda-toolkit
CUDNN: 9.1.1.17
NVIDIA GPU: GeForce RTX 3080
NVIDIA Driver Version: 535.183.01
Python Version: 3.12.9

Steps to reproduce

This bug can be reproduced by the following code with the model in the attachment.

from typing import Dict, List, Literal, Optional
import sys
import os

import numpy as np
import onnx
import onnxruntime
from onnx import ModelProto, TensorProto, helper, mapping

import pickle

def test():
    
    onnx_model = onnx.load("a1475.onnx")
    print(onnx_model.opset_import[0].version)

    with open("inputs.pkl", "rb") as fp:
        inputs = pickle.load(fp)

    ort_session = onnxruntime.InferenceSession(
            onnx_model.SerializeToString(), providers=["CPUExecutionProvider"]
        )
    ort_output = ort_session.run([], inputs)
    
    print("ONNXRuntime:\n", ort_output)
    
    #--------------------------------------------
    
    ort_session = onnxruntime.InferenceSession(
            onnx_model.SerializeToString(), providers=["CUDAExecutionProvider"]
        )
    ort_output = ort_session.run([], inputs)
    
    print("ONNXRuntime:\n", ort_output)
    
if __name__ == "__main__":
    test()

testcase.zip

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.23.0.dev20250515001

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU, CUDA

Execution Provider Library Version

CUDA: cuda-12.2.2::cuda-toolkit CUDNN: 9.1.1.17

[tianleiwu]

Could you try build from source with --cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=1. Then you can set ORT_DEBUG_NODE_IO_DUMP_INPUT_DATA=1 and ORT_DEBUG_NODE_IO_DUMP_OUTPUT_DATA=1 before running test. It's able to see which node in CUDA provider generates NaN.

See https://onnxruntime.ai/docs/build/eps.html#cuda