Auto-TensorRT engine compilation, or improved documentation for it

[fxmarty]

Feature request

For decoder models with cache, it can be painful to manually compile the TensorRT engine as ONNX Runtime does not give options to specify shapes. The engine build could maybe be done automatically.

The current doc is only for use_cache=False, which is not very interesting. It could be improved to show how to pre-build the TRT with use_cache=True.

References:
https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/gpu#tensorrt-engine-build-and-warmup
microsoft/onnxruntime#13559

Motivation

TensorRT is fast

Your contribution

will work on it sometime

[fxmarty]

microsoft/onnxruntime#13851 will make this much easier

[puyuanOT]

Is there any update on this? I also suffer from the difficulty to use TensorRT as the provider after run optimum-cli optimization.

[chilo-ms]

Hi,

1.15 ORT TRT supports explicit input shape meaning users can provide the shape range for all the dynamic shape input.
Please see the PR as well as the doc for usage/details.

Let us know if you have further questions or other feedbacks for ORT TRT. We are willing to make ORT TRT easier to use.

[fxmarty]

@chilo-ms Thanks a lot, that looks great! It is not at the top of our todo for now, but we're welcoming community contribution to interface well TensorrtExecutionProvider with ORTModel classes!