GlobalAveragePool outputs different results for the CPUExecutionProvider and CUDAExecutionProvider

[coffezhou]

Describe the issue

For the following onnx model,

the results are different for the CPUExecutionProvider and CUDAExecutionProvider.
For the CPUExecutionProvider, the results are as follows:

CPU_ONNXRuntime:
 [array([[[[-1.3999489e-06]],

        [[ 2.0466288e-07]],

        [[-2.0101362e-07]]]], dtype=float32)]

while the results for the CUDAExecutionProvider are as follows:

CUDA_ONNXRuntime:
 [array([[[[-6.4850461e-08]],

        [[-1.6953871e-08]],

        [[ 6.6409001e-08]]]], dtype=float32)]

Although the values in the above results are very small, it can affect the final result of the original model. I have checked that the results of
InstanceNormalization are identical for the CPUExecutionProvider and CUDAExecutionProvider.

To reproduce

Environment

OS: Ubuntu 20.04
onnxruntime: 1.23.0.dev20250626001
CUDA: cuda-12.2.2::cuda-toolkit
CUDNN: 9.1.1.17
NVIDIA GPU: GeForce RTX 3080
NVIDIA Driver Version: 535.183.01
Python Version: 3.12.9

Steps to reproduce

This bug can be reproduced by the following code with the model in the attachment.

from typing import Dict, List, Literal, Optional
import sys
import os

import numpy as np
import onnx
import onnxruntime
from onnx import ModelProto, TensorProto, helper, mapping

import pickle

def test():
    
    onnx_model = onnx.load("333.onnx")
    print(onnx_model.opset_import[0].version)

    with open("inputs.pkl", "rb") as fp:
        inputs = pickle.load(fp)

    ort_session = onnxruntime.InferenceSession(
            onnx_model.SerializeToString(), providers=["CPUExecutionProvider"]
        )
    cpu_ort_output = ort_session.run([], inputs)
    
    print("CPU_ONNXRuntime:\n", cpu_ort_output)
    
    #--------------------------------------------
    
    ort_session = onnxruntime.InferenceSession(
            onnx_model.SerializeToString(), providers=["CUDAExecutionProvider"]
        )
    cuda_ort_output = ort_session.run([], inputs)
    
    print("CUDA_ONNXRuntime:\n", cuda_ort_output)
    
    
if __name__ == "__main__":
    test()

testcase.zip
In testcase.zip, '444.onnx' is the model that removes the GlobalAveragePool operator for '333.onnx'. '444.onnx' is used to verify whether the results of InstanceNormalization for the CPUExecutionProvider and CUDAExecutionProvider are identical.

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.23.0.dev20250626001

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU, CUDA

Execution Provider Library Version

cuda-12.2.2::cuda-toolkit CUDNN: 9.1.1.17

[xadupre]

The number you shared are very close to zero. Since you normalize before GlobalAveragePool, it should be exactly zero. I did not check your inputs. I assume the order of magnitude for its elements is close 1. 1e-7 is below the precision float32 can offer.

[coffezhou]

The number you shared are very close to zero. Since you normalize before GlobalAveragePool, it should be exactly zero. I did not check your inputs. I assume the order of magnitude for its elements is close 1. 1e-7 is below the precision float32 can offer.

Thanks for your reply!
The mode in this issue is reduced from the following model:

With the same inputs, different results are produced for for the CPUExecutionProvider and CUDAExecutionProvider.
CPUExecutionProvider:

 [array([[[[ 0.7753263 ]],

        [[-0.11334736]],

        [[ 0.11132631]]]], dtype=float32)]

CUDAExecutionProvider:

 [array([[[[ 0.4375481]],

        [[ 0.1143883]],

        [[-0.4480636]]]], dtype=float32)]

There are obvious differences in the above results. After some analysis, I finally locate that GlobalAveragePool produces different results.

The following attachment is the original model.
111.zip