[Discussion] Performance & Architectural Differences: ONNX Runtime (TensorRT EP) vs. Native TensorRT Backend

[RodrickSia]

Title Suggestion:

[Discussion] Performance Comparison: ONNX Runtime (TensorRT Accelerator) vs. Native TensorRT

---

Hey everyone,

I am looking into deploying an AI model on an NVIDIA GPU and want to understand the exact performance differences between two setups:

1. Running the model via ONNX Runtime with the TensorRT execution provider enabled.
2. Running the model natively using NVIDIA TensorRT directly (e.g., using trtexec).

Since both setups use the same TensorRT engine under the hood to speed up the model, I'd love to know what the real-world performance gap looks like.

• Latency & Speed: Does the extra software layer in ONNX Runtime introduce any noticeable delay or overhead compared to running raw Native TensorRT?
• Throughput: In high-traffic production environments, is there a significant difference in how many frames or requests per second each setup can handle?
• Resource Usage: Is there any major difference in GPU memory (VRAM) usage or CPU overhead between the two?

If anyone has run direct speed tests or benchmarks comparing these two approaches, please share your numbers and experiences below!