❓ [Question] How can I load a TensorRT model generated with `trtexec`?

[javiabellan]

❓ Question

How can I load into Pytorch a TensorRT model engine (.trt or .plan) generated with trtexec ?

I have the following TensorRT model engine (generated from a ONNX file) using the trtexec tool provided by Nvidia

trtexec --onnx=../2.\ ONNX/CLIP-B32-image.onnx \
        --saveEngine=../4.\ TensorRT/CLIP-B32-image.trt \
        --minShapes=input:1x3x224x224 \
        --optShapes=input:1x3x224x224 \
        --maxShapes=input:32x3x224x224 \
        --fp16

I want to load it into Pytorch for using the Pytorch's dataloader for fast batch ineference.

[peri044]

You can use torch_tensorrt.ts.embed_engine_in_new_module API to do this. Reference: https://pytorch.org/TensorRT/py_api/ts.html?highlight=embed#torch_tensorrt.ts.embed_engine_in_new_module

The resulting torchscript module can be used with the dataloaders for inference.

[javiabellan]

Thanks, that's exactly what I needed.