ONNX Execution Providers

ONNX Execution Providers

Section: AI & Inference · Inference-Routing-and-Planning · Home · Model-Manifests-and-Governance

ONNX EPs abstract hardware for ASR, translation, TTS, and related stages. Strategy: TensorRT RTX ABI plugin for NVIDIA when available, Windows ML for catalog EPs, DirectML as legacy GPU fallback, CPU otherwise. Registration alone is never readiness.

Architecture and Registration

The system uses a registry-based architecture to manage execution provider bootstrappers. These bootstrappers are responsible for the specialized setup required for each hardware backend before an ONNX Runtime session is initialized.

flowchart TD
    Registry[OnnxExecutionProviderBootstrapperRegistry]
    Factory[ExecutionProviderFactory]
    TRT_RTX[TensorRtRtxProviderBootstrapFactory]
    WinML[WinMlCatalogProvider]
    
    Registry -->|Contains| TRT_RTX
    Registry -->|Contains| WinML
    Factory -->|Requests| Registry
    Registry -->|Resolves| Bootstrapper

Loading

The registry serves as a central hub for all ONNX-specific hardware adapters.

Bootstrapper Registry

The OnnxExecutionProviderBootstrapperRegistry is a sealed class that holds a collection of IOnnxExecutionProviderBootstrapper implementations. It provides methods to register new bootstrappers and retrieve them by their provider ID, such as "trt-rtx" or "windows-ml".

TensorRT RTX Bootstrap

The TensorRT RTX provider is handled as a standalone ONNX Runtime EP ABI plugin (NvTensorRTRTXExecutionProvider). Its bootstrap process is managed by the TensorRtRtxProviderBootstrapFactory, which involves:

1. Validating a specialized DLL bundle.
2. Registering the standalone plugin from the validated bundle.
3. Appending the execution provider to the OrtEnv via AppendExecutionProvider.

This provider is not a standard Windows ML catalog EP but a dedicated high-performance plugin.

Execution Provider Strategy

Trackdub classifies EPs into distinct categories based on their performance characteristics and platform support.

Provider ID	Implementation Type	Platform	Strategy Posture
trt-rtx	Standalone ABI Plugin	Windows (NVIDIA)	Primary high-performance GPU route.
windows-ml	Windows ML Integration	Windows	Broad surface for DirectML and Catalog EPs.
directml	DML through WinML	Windows	Legacy GPU fallback.
coreml	CoreML EP	macOS (Apple Silicon)	Primary Apple GPU route.
cpu	Native ONNX Runtime	All	Terminal fallback.

Windows ML Catalog EPs

For Windows, the application leverages Windows ML to access a variety of certified hardware-specific execution providers. These are identified by specific constant strings used during registration.

• MIGraphX: Used for AMD hardware acceleration.
• OpenVINO: Used for Intel hardware optimization.
• QNN / VitisAI: Targeted for NPU and specialized AI accelerators.

Runtime Planning and Readiness

A critical rule in the Trackdub architecture is No Fake Readiness. The state of an Execution Provider is tracked through multiple granular phases to ensure stability before inference begins.

sequenceDiagram
    participant Planner as RuntimePlanner
    participant Registry as BootstrapperRegistry
    participant Hardware as HardwareProbes
    
    Planner->>Registry: GetProvider(providerId)
    Registry-->>Planner: Return Bootstrapper
    Planner->>Hardware: Check EP Availability
    Hardware-->>Planner: Available? (Yes/No)
    Note over Planner: Check Model compatibility in Manifest
    Planner->>Planner: Verify Runtime Installed/DLLs Present
    Note over Planner: EP Marked as Ready

Loading

Readiness is verified per model and per stage, distinguishing between a provider being "registered" and the hardware actually being capable of running the required ONNX graph.

Hardware Profiling

The system includes a HardwareProfiler that benchmarks these execution providers against actual pipeline workloads (VAD, ASR, TTS). The results are stored as BenchmarkEvidence to inform quality preset recommendations and identify driver or fingerprint invalidations.

Implementation Details

Provider Registration

Bootstrappers are registered during the application's composition phase, typically within Trackdub.Composition. They link the provider-neutral contracts in Trackdub.Inference to the concrete session wrappers in Trackdub.Inference.Onnx.

Summary of Configuration

Execution providers are documented in the bundled-models.manifest.json under expected_runtime. This allows the IRuntimePlanner to match specific models to their optimal hardware route (e.g., trt-rtx|windows-ml|onnxruntime-directml).