-
Notifications
You must be signed in to change notification settings - Fork 0
DirectML and TensorRT
DirectML is the standard ONNX backend in the normal Neo package. It is intended to provide broad Windows GPU compatibility.
For most users, start with DirectML.
TensorRT is an optional backend for supported NVIDIA GeForce RTX GPUs.
Current Neo backend design targets:
- RTX 20 Series
- RTX 30 Series
- RTX 40 Series
- RTX 50 Series
The TensorRT backend is distributed separately so that the normal Neo package can remain small and simple.
Place the separately distributed TensorRT backend files in:
backends/tensorrt/
A valid pack contains backend.json and the required TensorRT / CUDA provider components.
TensorRT engines are generated on the user's PC for a specific combination of GPU, model, resolution, and profile. The first run can therefore take longer.
Do not assume an engine generated for one GPU can be copied to a different GPU generation.
Models that cannot use the TensorRT path may use DirectML instead, depending on the current Neo build and model compatibility.
Use DirectML when compatibility and simplicity are most important.
Try TensorRT when you have a supported NVIDIA RTX GPU and need more ONNX performance.
| Filter | Input | DirectML | TensorRT | Speedup |
|---|---|---|---|---|
| AnimeJaNai HD V3.1 Performance | 854×480 | ~3.83 ms | ~2.33 ms | ~1.64× |
| AnimeJaNai SD V1 beta34 Compact | 640×480 | ~10.58 ms | ~4.70 ms | ~2.25× |
| RIFE v4.22 Lite | 1280×720 | ~28.21 ms | ~12.36 ms | ~2.28× |