Keep falling into CPU Path

[NiksterX]

Describe the issue

Hello, i've satisfied (allegedly) all the requirements needed, my GPU supports up to CUDA 13, I have CUDA 12.8 installed (Tried 12.2 and 12.6) i'm using cuDNN v9.7 also tried v8.9, the correct GPU supported ONNX runtime package but can't seem to get GPU access. The runtime/cuda/shared DLLs are present in project directory.

To reproduce

00000001 0.00000000 [10540] ORT:CPU
00000002 0.00018820 [10540] Failed to load onnxruntime_providers_cuda.dll, error code: 1114
00000003 0.00029480 [10540] cudart64_12.dll loaded successfully
00000004 0.00085590 [10540] CUDA not available (missing: provider), using CPU
00000005 0.49468371 [10540] ORT:CPU initialized

Urgency

No response

Platform

Windows

OS Version

Windows 11 24h2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-win-x64-gpu-1.23.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.8

[snnn]

Try this:
#9630 (comment)

[NiksterX]

Try this: #9630 (comment)

Thanks but the exe calls a .DLL which the main code inhabits and in turn calls the .onnx, tried DependenciesGui the list looks clean no missing DLLs all requirements are complete and yet only CPU path is viable which isn't a decent option for me given it's a real-time app and desperately need GPU compute to work. Will keep digging around.

Got the GPU path but unlocked a new monster level:

00000001	0.00000000	[4912] ORT:CPU	
00000002	0.00021420	[4912] CUDA EP enabled successfully	
00000003	0.50153577	[4912] ORT:GPU initialized	
00000004	0.50167090	[4912] Inference error: Non-zero status code returned while running Conv node. Name:'/model.0/conv/Conv' Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device	

[NiksterX]

Posting this for time travelers.

DirectML GPU path works but with undesirable latency of 265ms, (Surprisingly CPU Path latency is 18ms) tried building targeted DLLs for my own nVidia GPU specifically and that failed too (Same error message as the first post).

This might be helpful for some: https://nietras.com/2021/01/25/onnxruntime/

EDIT:

NVM DirectML shows promising results, it was poor coding on my end.

Warmup run 1: 203.59 ms (initial GPU spin-up)
Warmup run 2: 9.90 ms
Warmup run 3: 12.16 ms

Real average latency: 11.03 ms
Real throughput: 90.7 FPS

[ThreeDeeJay]

Got the GPU path but unlocked a new monster level:

NiksterX How did you manage to get it working?
I tried copying cuDNN and TensorRT DLLs into CUDA's DLL folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\bin\x64 (which was added to PATH by the installer)

But it still falls back to CPU with error 1114.

My use case is a DirectShow plugin to convert 2D videos to 3D by using depth estimation model to reproject the source frames into left and right views. https://github.com/ThreeDeeJay/3Deflatten/releases/download/latest-main/3Deflatten.zip
If you wanna try it, I'd recommend PotPlayer 64-bit > right click main window > Preferences > Filter Control > Filter Priority (Overall) > Add external filter > locate Win64_GPU/3Deflatten_x64.ax, check it > Priority > Prefer > Close PotPlayer > Download AI model via > Setup.py > Start PotPlayer and play a video. Logging can be enabled by setting env var DEFLATTEN_LOG_FILE=C:\path\to\deflatten.log

By the way, I'm running Windows 11 23H2 with 595.71 on a 2080 Ti, and I also ran DLLs through Dependencies and didn't see any DLL was missing.
There's also Win64/3Deflatten_x64.ax which uses DirectML but performance is still poor (~500ms per frame, so 2FPS) even after leaving it running for a couple minutes. Not sure if I should keep waiting, though the log said all shaders were generated. 🤔

[NiksterX]

Got the GPU path but unlocked a new monster level:

@NiksterX How did you manage to get it working? I tried copying cuDNN and TensorRT DLLs into CUDA's DLL folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\bin\x64 (which was added to PATH by the installer) But it still falls back to CPU with error 1114.

My use case is a DirectShow plugin to convert 2D videos to 3D by using depth estimation model to reproject the source frames into left and right views. https://github.com/ThreeDeeJay/3Deflatten/releases/download/latest-main/3Deflatten.zip If you wanna try it, I'd recommend PotPlayer 64-bit > right click main window > Preferences > Filter Control > Filter Priority (Overall) > Add external filter > locate Win64_GPU/3Deflatten_x64.ax, check it > Priority > Prefer > Close PotPlayer > Download AI model via > Setup.py > Start PotPlayer and play a video. Logging can be enabled by setting env var DEFLATTEN_LOG_FILE=C:\path\to\deflatten.log

By the way, I'm running Windows 11 23H2 with 595.71 on a 2080 Ti, and I also ran DLLs through Dependencies and didn't see any DLL was missing. There's also Win64/3Deflatten_x64.ax which uses DirectML but performance is still poor (~500ms per frame, so 2FPS) even after leaving it running for a couple minutes. Not sure if I should keep waiting, though the log said all shaders were generated. 🤔

This isn't my code but try to run this python script and see if you're missing any dependencies or going through versions mismatch.

import sys
import subprocess
import importlib
import tkinter as tk
from tkinter import ttk, scrolledtext, messagebox
import threading
import webbrowser
import re

# ---------------- Helper Functions ----------------
def run_command(cmd):
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return result.stdout.strip()
    except:
        return None

def log(msg, text_widget):
    text_widget.insert(tk.END, msg + "\n")
    text_widget.see(tk.END)

def check_package_version(package_name, log_widget):
    try:
        module = importlib.import_module(package_name)
        log(f"{package_name} installed: version {module.__version__}", log_widget)
        return module
    except ImportError:
        log(f"{package_name} not installed.", log_widget)
        return None

def get_system_cuda_version():
    output = run_command(["nvcc", "--version"])
    if output:
        match = re.search(r"release (\d+\.\d+)", output)
        if match:
            return match.group(1)
    return None

def get_gpu_info():
    output = run_command(["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"])
    if output:
        return output.splitlines()[0].strip()
    return None

def recommend_cuda_for_gpu(gpu_name):
    # Simplified mapping: modern GPUs support latest CUDA
    return "12.1"

def get_latest_pytorch_version():
    # Latest stable PyTorch version
    return "2.2.0"

def get_pytorch_wheel_command(cuda_version, torch_version):
    base_command = f"torch=={torch_version} torchvision torchaudio"
    if cuda_version.startswith("12.1"):
        return f"{base_command} --index-url https://download.pytorch.org/whl/cu121"
    elif cuda_version.startswith("11.8"):
        return f"{base_command} --index-url https://download.pytorch.org/whl/cu118"
    else:
        # fallback CPU-only
        return base_command

def install_package(command, log_widget):
    log(f"Installing: pip install {command}", log_widget)
    subprocess.check_call([sys.executable, "-m", "pip", "install"] + command.split())

def check_gpu_with_torch(log_widget):
    try:
        import torch
        log(f"PyTorch imported successfully. Version: {torch.__version__}", log_widget)
        if torch.cuda.is_available():
            log(f"GPU detected by PyTorch: {torch.cuda.get_device_name(0)}", log_widget)
        else:
            log("PyTorch installed but GPU not detected.", log_widget)
    except Exception as e:
        log(f"Error importing PyTorch: {e}", log_widget)

# ---------------- Main Automation ----------------
def run_checks(progress_var, log_widget):
    total_steps = 5

    
    log("=== GPU Check ===", log_widget)
    gpu_name = get_gpu_info()
    if gpu_name:
        log(f"Detected GPU: {gpu_name}", log_widget)
        driver_output = run_command(["nvidia-smi"])
        log(f"NVIDIA driver info:\n{driver_output}", log_widget)
    else:
        log("No NVIDIA GPU detected or driver missing.", log_widget)
        messagebox.showwarning("GPU Missing", "No NVIDIA GPU detected or drivers missing.")
        progress_var.set(0)
        return
    progress_var.set(1/total_steps*100)

    
    log("\n=== Python Version Check ===", log_widget)
    major, minor = sys.version_info[:2]
    log(f"Detected Python version: {major}.{minor}", log_widget)
    if major < 3 or (major == 3 and minor < 9):
        log("Python < 3.9 detected.", log_widget)
        if messagebox.askyesno("Python Update Recommended",
                               "Python 3.9 or higher is recommended.\nOpen Python download page?"):
            webbrowser.open("https://www.python.org/ftp/python/3.12.0/python-3.12.0-amd64.exe")
        progress_var.set(0)
        return
    else:
        log("Python version is compatible.", log_widget)
    progress_var.set(2/total_steps*100)

    
    log("\n=== CUDA Check ===", log_widget)
    system_cuda = get_system_cuda_version()
    recommended_cuda = recommend_cuda_for_gpu(gpu_name)
    log(f"Detected CUDA version: {system_cuda}", log_widget)
    log(f"Recommended CUDA version for {gpu_name}: {recommended_cuda}", log_widget)
    if not system_cuda or float(system_cuda) < float(recommended_cuda):
        log("CUDA missing or version too old.", log_widget)
        if messagebox.askyesno("CUDA Required",
                               f"CUDA {recommended_cuda} is required for GPU.\nOpen CUDA download page?"):
            webbrowser.open("https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda_12.1.0_531.14_windows.exe")
        progress_var.set(0)
        return
    progress_var.set(3/total_steps*100)

    
    log("\n=== PyTorch Check ===", log_widget)
    torch_pkg = check_package_version("torch", log_widget)
    tv_pkg = check_package_version("torchvision", log_widget)
    needs_install = False

    if not torch_pkg or not tv_pkg:
        needs_install = True
    else:
        import torch
        if not torch.cuda.is_available():
            log("Installed PyTorch is CPU-only. GPU not enabled.", log_widget)
            needs_install = True

    if needs_install:
        torch_version = get_latest_pytorch_version()
        command = get_pytorch_wheel_command(recommended_cuda, torch_version)
        install_package(command, log_widget)
        check_package_version("torch", log_widget)
        check_package_version("torchvision", log_widget)
    progress_var.set(4/total_steps*100)

    
    log("\n=== GPU Validation Check ===", log_widget)
    check_gpu_with_torch(log_widget)
    progress_var.set(100)
    log("\n=== Setup Complete ===", log_widget)

# ---------------- GUI ----------------
def start_gui():
    root = tk.Tk()
    root.title("Model-Trainer-GPU-Setup")

    progress_var = tk.DoubleVar()
    progress = ttk.Progressbar(root, variable=progress_var, maximum=100)
    progress.pack(fill=tk.X, padx=10, pady=10)

    log_widget = scrolledtext.ScrolledText(root, width=100, height=25)
    log_widget.pack(padx=10, pady=10)

    def start_thread():
        threading.Thread(target=run_checks, args=(progress_var, log_widget), daemon=True).start()

    start_btn = tk.Button(root, text="Start GPU Setup", command=start_thread)
    start_btn.pack(pady=5)

    root.mainloop()

if __name__ == "__main__":
    start_gui()

[ThreeDeeJay]

It recommended CUDA 12.1, so I installed it (I even uninstalled the driver and installed the one bundled with that CUDA installer):

=== GPU Check ===
Detected GPU: NVIDIA GeForce RTX 2080 Ti
NVIDIA driver info:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 531.14                 Driver Version: 531.14       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                      TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti    WDDM | 00000000:01:00.0  On |                  N/A |
|  0%   31C    P0               58W / 260W|    445MiB / 11264MiB |     25%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3216    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A      9208    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A     11744    C+G   ...5n1h2txyewy\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     12368    C+G   ...CBS_cw5n1h2txyewy\TextInputHost.exe    N/A      |
+---------------------------------------------------------------------------------------+

=== Python Version Check ===
Detected Python version: 3.13
Python version is compatible.

=== CUDA Check ===
Detected CUDA version: 12.1
Recommended CUDA version for NVIDIA GeForce RTX 2080 Ti: 12.1

=== PyTorch Check ===
torch not installed.
torchvision not installed.
Installing: pip install torch==2.2.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

But it failed installing torch:

Looking in indexes: https://download.pytorch.org/whl/cu121
ERROR: Could not find a version that satisfies the requirement torch==2.2.0 (from versions: none)

So I browsed that index-url and noticed there's no torch 2.2.0 for Python 3.12. Only these would apply to my case
torch-2.2.0+cu121-cp38-cp38-win_amd64.whl
torch-2.2.0+cu121-cp39-cp39-win_amd64.whl
torch-2.2.0+cu121-cp310-cp310-win_amd64.whl
torch-2.2.0+cu121-cp311-cp311-win_amd64.whl
torch-2.2.0+cu121-cp312-cp312-win_amd64.whl
so I downgraded to 3.9.6 where I know CUDA worked in a similar app and that went through.

=== GPU Check ===
Detected GPU: NVIDIA GeForce RTX 2080 Ti
NVIDIA driver info:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 531.14                 Driver Version: 531.14       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                      TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti    WDDM | 00000000:01:00.0  On |                  N/A |
|  0%   29C    P8               27W / 260W|    488MiB / 11264MiB |     24%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3216    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A      9208    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A     11744    C+G   ...5n1h2txyewy\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     12368    C+G   ...CBS_cw5n1h2txyewy\TextInputHost.exe    N/A      |
+---------------------------------------------------------------------------------------+

=== Python Version Check ===
Detected Python version: 3.9
Python version is compatible.

=== CUDA Check ===
Detected CUDA version: 12.1
Recommended CUDA version for NVIDIA GeForce RTX 2080 Ti: 12.1

=== PyTorch Check ===
torch installed: version 2.2.0+cu121
torchvision installed: version 0.17.0+cu121

=== GPU Validation Check ===
PyTorch imported successfully. Version: 2.2.0+cu121
GPU detected by PyTorch: NVIDIA GeForce RTX 2080 Ti

=== Setup Complete ===

However, the console reported this error:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "%LOCALAPPDATA%\3Deflatten\dlcache\depcheck.py", line 128, in run_checks
    torch_pkg = check_package_version("torch", log_widget)
  File "%LOCALAPPDATA%\3Deflatten\dlcache\depcheck.py", line 24, in check_package_version
    module = importlib.import_module(package_name)
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\__init__.py", line 1471, in <module>
    from .functional import *  # noqa: F403
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\functional.py", line 9, in <module>
    import torch.nn.functional as F
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\nn\__init__.py", line 1, in <module>
    from .modules import *  # noqa: F403
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\nn\modules\__init__.py", line 35, in <module>
    from .transformer import TransformerEncoder, TransformerDecoder, \
  File "%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\nn\modules\transformer.py", line 20, in <module>
    device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
%LOCALAPPDATA%\Programs\Python\Python39\lib\site-packages\torch\nn\modules\transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),

But this seemed to fix it:

pip uninstall NumPy
pip install NumPy==1.26.4

Anyhow, looking at that python script I noticed 12.1 is hard-coded so maybe that's just the latest version at the time the script was made? Because the 2080 Ti (Turing) should also support CUDA 13 https://en.wikipedia.org/wiki/CUDA#GPUs_supported

Anyhow, are python modules like torch even required for native C++ plugins?
If so, is anything else required for onnxruntime with CUDA besides cuDNN and TensorRT (which is the one I want to use)?

[NiksterX]

No, torch isn't required for native C++ plugins. ONNX Runtime is a standalone inference engine, so if you're loading an ONNX model directly there is no PyTorch dependency in the runtime stack.

For the CUDA execution provider you typically just need CUDA and cuDNN to match, this part is very critical, they need to be a match.

If you're using the TensorRT execution provider instead, then the stack becomes CUDA + cuDNN + TensorRT, with ONNX Runtime built with TensorRT support.

[ThreeDeeJay]

I used CUDA Toolkit 11.7.0 (May 2022) with cuDNN v8.9.3 (July 11th, 2023), for CUDA 11.x, a combination which works in the other app I mentioned which uses pytorch

I also added TensorRT 10.13 GA for Windows 10, 11, Server 2022 and CUDA 11.0 to 11.8 ZIP Package (last 11.x build) as well as TensorRT-10.0.0.6.Windows10.win10.cuda-11.8 but it doesn't even work either. So I added a dependency scanner to my plugin's log:

[INFO ] --- CUDA/TRT dependency scan ---
[INFO ]   ORT 1.16.3
[INFO ]     CUDA 11.x  (cudart64_110.dll)
[INFO ]     cuDNN 8.x  (cudnn64_8.dll + split infer/train DLLs)
[INFO ]     TensorRT 10.13.0.35 (CUDA 11.8 build)
[INFO ]   [OK]     nvcuda.dll  (NVIDIA driver kernel proxy -- must be in System32)
[INFO ]   [OK]     cudart64_110.dll  (CUDA 11.x runtime)
[INFO ]   [OK]     cublas64_11.dll  (cuBLAS 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cublasLt64_11.dll  (cuBLAS-Lt 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cufft64_10.dll  (cuFFT 10 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cusolver64_11.dll  (cuSolver 11 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     curand64_10.dll  (cuRand 10 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     cudnn64_8.dll  (cuDNN 8.x main library)
[INFO ]   TensorRT 10.13.0.x libraries (CUDA 11.8 build):
[INFO ]   [OK]     nvinfer_10.dll  (TensorRT 10 inference engine)
[INFO ]   [OK]     nvonnxparser_10.dll  (TensorRT 10 ONNX parser)
[INFO ]   [OK]     nvinfer_dispatch_10.dll  (TRT 10 dispatch runtime)
[INFO ]   [OK]     nvinfer_lean_10.dll  (TRT 10 lean runtime)
[INFO ]   [OK]     nvinfer_plugin_10.dll  (TRT 10 plugins)
[INFO ] --- end dependency scan ---
[WARN ] onnxruntime_providers_tensorrt.dll exists but failed to load (error 126 = missing dependency).
[INFO ]   [dep-scan] Direct imports of: C:\Programs\3Deflatten\Release\Win64_GPU\onnxruntime_providers_tensorrt.dll
[INFO ]   [dep-scan]   [OK]      cudnn64_8.dll
[INFO ]   [dep-scan]   [OK]      cublas64_11.dll
[WARN ]   [dep-scan]   [MISSING] nvinfer.dll  (error=126)
[WARN ]   [dep-scan]   [MISSING] nvinfer_plugin.dll  (error=126)
[WARN ]   [dep-scan]   [MISSING] nvonnxparser.dll  (error=126)
[INFO ]   [dep-scan]   [OK]      cudart64_110.dll
[INFO ]   [dep-scan]   [OK]      onnxruntime_providers_shared.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-file-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-errorhandling-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processenvironment-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-debug-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      MSVCP140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140_1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-runtime-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-heap-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-convert-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-stdio-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-filesystem-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-time-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-math-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-locale-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-environment-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-heap-l2-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-localization-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-rtlsupport-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-profile-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-sysinfo-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-interlocked-l1-1-0.dll
[INFO ]   [dep-scan] Summary: 34 OK,  3 MISSING
[INFO ] --- CUDA/TRT dependency scan ---
[INFO ]   ORT 1.16.3:
[INFO ]     CUDA 11.x  (cudart64_110.dll)
[INFO ]     cuDNN 8.x  (cudnn64_8.dll + split infer/train DLLs)
[INFO ]   [OK]     nvcuda.dll  (NVIDIA driver kernel proxy -- must be in System32)
[INFO ]   [OK]     cudart64_110.dll  (CUDA 11.x runtime)
[INFO ]   [OK]     cublas64_11.dll  (cuBLAS 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cublasLt64_11.dll  (cuBLAS-Lt 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cufft64_10.dll  (cuFFT 10 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cusolver64_11.dll  (cuSolver 11 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     curand64_10.dll  (cuRand 10 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     cudnn64_8.dll  (cuDNN 8.x main library)
[INFO ] --- end dependency scan ---
[WARN ] onnxruntime_providers_cuda.dll load failed error=1114 (DLL init failed -- all deps present).
[WARN ]   Running PE import scan to confirm no hidden missing dep:
[INFO ]   [dep-scan] Direct imports of: C:\Programs\3Deflatten\Release\Win64_GPU\onnxruntime_providers_cuda.dll
[INFO ]   [dep-scan]   [OK]      cublasLt64_11.dll
[INFO ]   [dep-scan]   [OK]      cublas64_11.dll
[INFO ]   [dep-scan]   [OK]      cudnn64_8.dll
[INFO ]   [dep-scan]   [OK]      cufft64_10.dll
[INFO ]   [dep-scan]   [OK]      onnxruntime_providers_shared.dll
[INFO ]   [dep-scan]   [OK]      cudart64_110.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-1.dll
[INFO ]   [dep-scan]   [OK]      MSVCP140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140_1.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-runtime-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-heap-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-stdio-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-math-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-interlocked-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-rtlsupport-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-errorhandling-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-debug-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-profile-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-sysinfo-l1-1-0.dll
[INFO ]   [dep-scan] Summary: 26 OK,  0 MISSING
[WARN ]   CUDA EP: DllMain failed.
[INFO ] Execution provider: CPU (auto fallback)

onnxruntime_providers_tensorrt.dll reports nvinfer.dll nvinfer_plugin.dll and nvonnxparser.dll are missing but they don't exist in TensorRT-10.13.0.35\lib. So I just tried copying and renaming nvinfer_10.dll nvinfer_plugin_10.dll and nvonnxparser_10.dll, which then were detected as OK by dep-scan but it still didn't work.
And onnxruntime_providers_cuda.dll reports all DLLs are present out of the box (after installing CUDA and cuDNN, or just copying the DLL to my plugin's DLL folder) but still fails.

According to #21769, the CUDA 11.7 + cuDNN 8.9 combo is compatible with ONNX Runtime 1.18.0/1.18.1, which I tried downgrading to it (gpu variant, and not cuda12 of course) in my plugin and the only difference I could notice in the logs is that they already load *_10.dll so I don't have to rename the 3 I mentioned, but still fallsback to CPU. 😔

[NiksterX]

I used CUDA Toolkit 11.7.0 (May 2022) with cuDNN v8.9.3 (July 11th, 2023), for CUDA 11.x, a combination which works in the other app I mentioned which uses pytorch

I also added [TensorRT 10.13 GA for Windows 10, 11, Server 2022 and CUDA 11.0 to 11.8 ZIP Package](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.13.0/zip/TensorRT-10.13.0.35.Windows.win10.cuda-11.8.zip) (last 11.x build) as well as [TensorRT-10.0.0.6.Windows10.win10.cuda-11.8](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.0/zip/TensorRT-10.0.0.6.Windows10.win10.cuda-11.8.zip) but it doesn't even work either. So I added a dependency scanner to my plugin's log:

[INFO ] --- CUDA/TRT dependency scan ---
[INFO ]   ORT 1.16.3
[INFO ]     CUDA 11.x  (cudart64_110.dll)
[INFO ]     cuDNN 8.x  (cudnn64_8.dll + split infer/train DLLs)
[INFO ]     TensorRT 10.13.0.35 (CUDA 11.8 build)
[INFO ]   [OK]     nvcuda.dll  (NVIDIA driver kernel proxy -- must be in System32)
[INFO ]   [OK]     cudart64_110.dll  (CUDA 11.x runtime)
[INFO ]   [OK]     cublas64_11.dll  (cuBLAS 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cublasLt64_11.dll  (cuBLAS-Lt 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cufft64_10.dll  (cuFFT 10 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cusolver64_11.dll  (cuSolver 11 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     curand64_10.dll  (cuRand 10 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     cudnn64_8.dll  (cuDNN 8.x main library)
[INFO ]   TensorRT 10.13.0.x libraries (CUDA 11.8 build):
[INFO ]   [OK]     nvinfer_10.dll  (TensorRT 10 inference engine)
[INFO ]   [OK]     nvonnxparser_10.dll  (TensorRT 10 ONNX parser)
[INFO ]   [OK]     nvinfer_dispatch_10.dll  (TRT 10 dispatch runtime)
[INFO ]   [OK]     nvinfer_lean_10.dll  (TRT 10 lean runtime)
[INFO ]   [OK]     nvinfer_plugin_10.dll  (TRT 10 plugins)
[INFO ] --- end dependency scan ---
[WARN ] onnxruntime_providers_tensorrt.dll exists but failed to load (error 126 = missing dependency).
[INFO ]   [dep-scan] Direct imports of: C:\Programs\3Deflatten\Release\Win64_GPU\onnxruntime_providers_tensorrt.dll
[INFO ]   [dep-scan]   [OK]      cudnn64_8.dll
[INFO ]   [dep-scan]   [OK]      cublas64_11.dll
[WARN ]   [dep-scan]   [MISSING] nvinfer.dll  (error=126)
[WARN ]   [dep-scan]   [MISSING] nvinfer_plugin.dll  (error=126)
[WARN ]   [dep-scan]   [MISSING] nvonnxparser.dll  (error=126)
[INFO ]   [dep-scan]   [OK]      cudart64_110.dll
[INFO ]   [dep-scan]   [OK]      onnxruntime_providers_shared.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-file-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-errorhandling-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processenvironment-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-debug-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      MSVCP140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140_1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-runtime-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-heap-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-convert-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-stdio-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-filesystem-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-time-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-math-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-locale-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-environment-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-heap-l2-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-localization-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-rtlsupport-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-profile-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-sysinfo-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-interlocked-l1-1-0.dll
[INFO ]   [dep-scan] Summary: 34 OK,  3 MISSING
[INFO ] --- CUDA/TRT dependency scan ---
[INFO ]   ORT 1.16.3:
[INFO ]     CUDA 11.x  (cudart64_110.dll)
[INFO ]     cuDNN 8.x  (cudnn64_8.dll + split infer/train DLLs)
[INFO ]   [OK]     nvcuda.dll  (NVIDIA driver kernel proxy -- must be in System32)
[INFO ]   [OK]     cudart64_110.dll  (CUDA 11.x runtime)
[INFO ]   [OK]     cublas64_11.dll  (cuBLAS 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cublasLt64_11.dll  (cuBLAS-Lt 11 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cufft64_10.dll  (cuFFT 10 -- in CUDA Toolkit bin)
[INFO ]   [OK]     cusolver64_11.dll  (cuSolver 11 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     curand64_10.dll  (cuRand 10 -- loaded by ORT CUDA EP at startup)
[INFO ]   [OK]     cudnn64_8.dll  (cuDNN 8.x main library)
[INFO ] --- end dependency scan ---
[WARN ] onnxruntime_providers_cuda.dll load failed error=1114 (DLL init failed -- all deps present).
[WARN ]   Running PE import scan to confirm no hidden missing dep:
[INFO ]   [dep-scan] Direct imports of: C:\Programs\3Deflatten\Release\Win64_GPU\onnxruntime_providers_cuda.dll
[INFO ]   [dep-scan]   [OK]      cublasLt64_11.dll
[INFO ]   [dep-scan]   [OK]      cublas64_11.dll
[INFO ]   [dep-scan]   [OK]      cudnn64_8.dll
[INFO ]   [dep-scan]   [OK]      cufft64_10.dll
[INFO ]   [dep-scan]   [OK]      onnxruntime_providers_shared.dll
[INFO ]   [dep-scan]   [OK]      cudart64_110.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-libraryloader-l1-2-1.dll
[INFO ]   [dep-scan]   [OK]      MSVCP140.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140_1.dll
[INFO ]   [dep-scan]   [OK]      VCRUNTIME140.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-runtime-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-heap-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-stdio-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-string-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-crt-math-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-synch-l1-2-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-interlocked-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-rtlsupport-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-errorhandling-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-processthreads-l1-1-1.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-debug-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-profile-l1-1-0.dll
[INFO ]   [dep-scan]   [OK]      api-ms-win-core-sysinfo-l1-1-0.dll
[INFO ]   [dep-scan] Summary: 26 OK,  0 MISSING
[WARN ]   CUDA EP: DllMain failed.
[INFO ] Execution provider: CPU (auto fallback)

onnxruntime_providers_tensorrt.dll reports nvinfer.dll nvinfer_plugin.dll and nvonnxparser.dll are missing but they don't exist in TensorRT-10.13.0.35\lib. So I just tried copying and renaming nvinfer_10.dll nvinfer_plugin_10.dll and nvonnxparser_10.dll, which then were detected as OK by dep-scan but it still didn't work. And onnxruntime_providers_cuda.dll reports all DLLs are present out of the box (after installing CUDA and cuDNN, or just copying the DLL to my plugin's DLL folder) but still fails.

According to #21769, the CUDA 11.7 + cuDNN 8.9 combo is compatible with ONNX Runtime 1.18.0/1.18.1, which I tried downgrading to it (gpu variant, and not cuda12 of course) in my plugin and the only difference I could notice in the logs is that they already load *_10.dll so I don't have to rename the 3 I mentioned, but still fallsback to CPU. 😔

Can you show me the output when you run this .py script

import onnxruntime as ort

print(ort.get_available_providers())

Or

import onnxruntime as ort

print("ORT version:", ort.__version__)
print("Available providers:", ort.get_available_providers())
print("Device:", ort.get_device())

[ThreeDeeJay]

Sorry for the late response. But I just got a chance to test it and I get:

ORT version: 1.19.2
Available providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
Device: GPU

I've been mix and matching so many versions that I don't even remember if, how or why I installed 1.19.2 in Python since I've mostly been using C++. Maybe ORT was just installed as a dependency of DepthViewer (the app where CUDA EP does work, on 11.7)

And to make things even weirder, ONNX Runtime v1.19.0 release notes say:

Default GPU packages use CUDA 12.x and Cudnn 9.x (previously CUDA 11.x/CuDNN 8.x) CUDA 11.x/CuDNN 8.x packages are moved to the aiinfra VS feed.

So if I understand correctly, ORT 1.19.2 is supposed to expect/require CUDA 12.
But I currently have installed 11.8 (with the driver it included)

Sat Mar 21 03:10:36 2026
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 522.06       Driver Version: 522.06       CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0  On |                  N/A |
| 15%   35C    P0    58W / 260W |    592MiB / 11264MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      8984    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     12416    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     14808    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A     16768    C+G   ....win64\steamwebhelper.exe    N/A      |
+-----------------------------------------------------------------------------+

So my only guess is that ORT works fine and is a lot more flexible on python, but there's just something I'm missing or there's a bug in my C++ plugin.

In the meantime, I've been using DirectML which performs a lot better than CPU, but still not good enough for real-time 1080p24 inferrence (I just get ~13FPS tops). 😔

[NiksterX]

13 FPS is rough and not usable, what's your model size? Try something lower than 640x640, but even then 13 FPS means something s broken, i used to get 90+ FPS on GTX 1060 with DML. If you're on the DML path you'll know when the model is running and the GPU usage increases.
You can check your latency with this script.

import torch
import torch_directml # Import the DirectML package
import time

# --- Configuration ---
# 1024x1024 matrix size for the test
MATRIX_SIZE = 1024
# Number of times to run the test for averaging
NUM_RUNS = 1000
# ---------------------

try:
    # 1. Initialize the DirectML device
    # This function automatically selects the default DirectML GPU device
    device = torch_directml.device()
    print(f"✅ Using DirectML device: {device}")

except Exception as e:
    print("❌ DirectML Initialization Failed.")
    print("Please ensure you have installed the 'torch-directml' package:")
    print("pip install torch-directml")
    print(f"Error details: {e}")
    exit()


# 2. Warm up (GPU driver lazy initialization)
# Create tensors and move them to the DirectML device
x = torch.randn(MATRIX_SIZE, MATRIX_SIZE, device=device)

print("Starting warm-up phase (10 matrix multiplications)...")
for _ in range(10):
    # Perform a matrix multiplication on the DirectML GPU
    y = torch.matmul(x, x)

# DirectML does not have a public equivalent to torch.cuda.synchronize() 
# that is simple to use for timing, so we will rely on the timing loop 
# and trust the DirectML backend to provide a reasonable measurement.

# 3. Start timing the main operation
print(f"Starting main latency test ({NUM_RUNS} runs)...")
start = time.time()

for _ in range(NUM_RUNS):
    # Matrix multiplication: This is the operation being timed
    y = torch.matmul(x, x)

end = time.time()

# 4. Calculate and print results
avg_latency_ms = (end - start) * 1000 / NUM_RUNS
print("-" * 50)
print(f"Average DirectML latency per {MATRIX_SIZE}x{MATRIX_SIZE} matmul:")
print(f"{avg_latency_ms:.3f} ms")
print("-" * 50)

# Optional: kernel launch latency (tiny op)
# NOTE: This measurement may be less accurate on DirectML compared to CUDA 
# due to the lack of a simple, explicit synchronization primitive for timing.

# start = time.time()
# for _ in range(NUM_RUNS):
#     y = x + 1
# end = time.time()
# print(f"Kernel launch latency (x+1): {(end - start)*1000/NUM_RUNS:.6f} ms") 

[ThreeDeeJay]

13 FPS is rough and not usable, what's your model size?

This one: https://huggingface.co/AXERA-TECH/Depth-Anything-3/resolve/main/models-ax650/da3-small.onnx
I'm using it to estimate the depth map to reproject 1080p24 videos into left and right views, where I get ~75ms inferences with DML
Though that was with an an old driver (426.06, which I use for full 3D Vision support)
I tested again after updating and now I'm getting ~37ms at ~50% GPU which is ~27FPS which is technically just about good enough for movies.
But that's just with the small model which outputs a 504x280 depth map, so I'd like to optimize my plugin as much as possible so we can afford using the bigger models for better 3D quality (ideally the depth map should match the RGB frame 1:1 but idk how feasible that'd be).

You can check your latency with this script.

✅ Using DirectML device: privateuseone:0
Starting warm-up phase (10 matrix multiplications)...
Starting main latency test (1000 runs)...
--------------------------------------------------
Average DirectML latency per 1024x1024 matmul:
0.374 ms
--------------------------------------------------

btw I needed downgrade to Python 3.10 so it lets me pip install torch-directml

On a side note, I found out that apparently my TensorRT RTX is compatible with my 2080 Ti despite some docs claiming otherwise.
So I compiled it here since it's not included in the official releases (yet?):
https://github.com/ThreeDeeJay/onnxruntime/releases/tag/latest-tensorrt-rtx
Though I'm yet to implement it into my plugin.