What the advantages of TRTorch?

[dancingpipi]

I used to use torch2trt to convert pytorch module, could you explain the advatage over torch2trt?

If the model contain op that tensorrt don't support, can trtorch convert it to engine?
Otherwise run the op supported by tensorrt with tensorrt, and other use libtorch?

I really appreciate for your great works, if you can answer my doubts, I will be very grateful.

[narendasan]

I think the main differences between torch2trt and TRTorch are its approach to conversion and its focus.

TRTorch is designed to be a robust path from PyTorch and TorchScript to TensorRT supporting C++ (via LibTorch) and eventually Python (via PyTorch) with the end goal being integrated into PyTorch itself as a inference backend for deployment scenarios.

torch2trt is great for experimentation and prototyping, it's lightweight and good for applications where the final application will remain in Python, it also currently has greater layer support.

Under the hood, TRTorch compiles stand alone torchscript code (no python dependency) to TensorRT and wraps it in a module, where as torch2trt monkey-patches PyTorch python functions to emit TensorRT layers when they are run, using that to construct the engine which is returned as a module.

In terms of advantages right now TRTorch is good if you want to deploy in C++ and torch2trt is good if you want to deploy in Python. In the future, hopefully you can do both in TRTorch without having to leave PyTorch in which case torch2trt is really useful in quickly adding extra capabilities that TRTorch doesn't have yet.

Right now our approach to unsupported TensorRT ops is to have users segment their models based on backend. So for example you may have your backbone running in TensorRT but later layers still running in LibTorch. You would have a module for each, compile the backbone then link to your later LibTorch layers. We are discussing other strategies for supporting unsupported TRT ops but right now this is the solution.

[dancingpipi]

Thanks for your reply!

Now I understand it, looking forward to your subsequent great work~