[multimodal] add tensorrt tutorial #2987
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Job PR-2987-3abbc5a is done. |
liangfu
added
the
model list checked
|
Job PR-2987-92bba50 is done. |
|
Job PR-2987-5cba8f8 is done. |
| "trt_predictor = MultiModalPredictor.load(path=model_path)\n", | ||
| "trt_predictor.optimize_for_inference()\n", | ||
| "\n", | ||
| "# Agagin, use first prediction for initialization (e.g., allocating memory)\n", |
There was a problem hiding this comment.
Maybe give more explanations of using the first prediction for initialization. Otherwise, users may wonder why this is necessary.
There was a problem hiding this comment.
Typo agagin?
Nice catch. Fixed typo.
Maybe give more explanations of using the first prediction for initialization. Otherwise, users may wonder why this is necessary.
This is indicated with an example being allocating memory
There was a problem hiding this comment.
The first time calling model forward takes more time than the following calls. I'm not sure whether we can explain this observation as model initialization since Pytorch uses dynamic graph and eager execution.
There was a problem hiding this comment.
What if the batch size is changeable during predictions?
There was a problem hiding this comment.
in my observation, it would automatically re-compile when batch_size dimension is larger than initialization.
There was a problem hiding this comment.
Any Pytorch documentation regarding this phenomenon?
There was a problem hiding this comment.
The re-compile only happens to tensorrt backend in onnxruntime, not directly related to pytorch.
This behavior is not well documented, it may related to Shape Inference for TensorRT Subgraphs
There was a problem hiding this comment.
The above pytorch module also uses the first prediction as initialization. That's why we use Again here. So, this behavior exists for both pytorch and onnxruntime/tensorrt?
There was a problem hiding this comment.
For pytorch, it is used for memory allocation; for onnxruntime/tensorrt, it is used for 1) fair comparison, 2) model compilation. (model compilation actually happens when calling optimize_for_inference() )
|
Job PR-2987-4a30526 is done. |
|
Job PR-2987-133854a is done. |
|
Job PR-2987-75ae324 is done. |
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.