Inference on a model prequantized in pytorch

[chrschinab]

Hallo,
I want to run inference on a resnet50 model prequantized to INT8 in pytorch . Is there a way to deploy this model in TensorRT? As far as I understood only a quantized ONNX model is accepted by TensorRT. However, currently, Pytorch does not support the conversion to quantized ONNX.
Is it possible to deploy a model prequantized in TFLite with TensorRT?

[mk-nvidia]

A future release of TensorRT will be able to import a prequantized model from PyTorch. These models can be quantized using the toolkit we released earlier.
We don’t yet support deploying a prequantized model from TFLite

[skyw]

Hi

Pytorch does support export (fake) quantized model to ONNX. Although per channel support hasn't been merged to master because it depends on ONNX opset13. See pytorch/pytorch#42835.