Explicit quantization in PyTorch before ONNX leads to slower TRT engine than ONNX PTQ

[liukang1811]

Describe the bug

Steps/Code to reproduce bug

Expected behavior

I’m experiencing unexpected behavior when performing PTQ (post-training quantization) using NVIDIA ModelOpt versus ONNX quantization. Here’s a comparison of two workflows:

1. Workflow 1 (trt1):

   • Perform PTQ on the PyTorch model (INT8) using NVIDIA ModelOpt.
   • Export the quantized PyTorch model to ONNX.
   • Convert the ONNX model to a TensorRT engine.
   • Observation: The TensorRT engine (trt1) contains many QuantizeLinear operators, and its inference time is 7.4 ms.

2. Workflow 2 (trt2):

   • Export the float PyTorch model directly to ONNX.
   • Apply PTQ (INT8) on the ONNX model.
   • Convert the quantized ONNX model to a TensorRT engine.
   • Observation: The TensorRT engine (trt2) has no QuantizeLinear operators, and its inference time is 5.9 ms.

So trt2 is about 20% faster than trt1. I suspect the presence of explicit quantization nodes in trt1 might be causing the slowdown. Is this expected behavior? Does doing PTQ in the ONNX space generate a more optimized engine compared to PTQ in the PyTorch space before ONNX export?

I checked the extra operators in trt1 compared to trt2, and they look like this:
"/image_encoder/image_encoder/_blocks_conv.21/_expand_conv/_expand_conv.0/input_quantizer/QuantizeLinear", "timeMs" : 3.38419, "averageMs" : 0.0129663, "medianMs" : 0.012448, "percentage" : 0.137381 }

System information

• Container used (if applicable): ?
• OS (e.g., Ubuntu 22.04, CentOS 7, Windows 10): ?
• CPU architecture (x86_64, aarch64): x86_64
• GPU name (e.g. H100, A100, L40S): A800
• GPU memory size: 80g
• Number of GPUs: 1
• Library versions (if applicable):
  • Python: 3.10
  • ModelOpt version or commit hash: 0.23
  • CUDA: 11.8
  • PyTorch: 2.1
  • Transformers: ?
  • TensorRT-LLM: ?
  • ONNXRuntime: 1.20
  • TensorRT: 10.7
• Any other details that may help: ?
  from tqdm import tqdm
  import modelopt.torch.quantization as mtq

def forward_loop(model):
...
model_outputs_iter = model(tensor_iamge)
quant_model = mtq.quantize(torch_model, mtq.INT8_DEFAULT_CFG, forward_loop)
with torch.no_grad():
torch.onnx.export(model, input_data, "./net_quant.onnx",
input_names= input_names, output_names =output, opset_version=17, verbose=False,)
model_onnx = onnx.load("./image_encoder_muti_input_ori_quant.onnx")
from modelopt.onnx.quantization.qdq_utils import qdq_to_dq
model_simp, check = simplify(model_onnx)
assert check, "Simplified ONNX model could not be validated"
model_dq = qdq_to_dq(model_simp)
onnx.save(model_dq, "./image_encoder_muti_input_simplified_quant.onnx")

Click to expand: Python script to automatically collect system information

</details>

[liukang1811]

@i-riyad “Could you help me answer this question? Thank you.”

[i-riyad]

This is expected. For convolutional models, ONNX PTQ generally performs better for TensorRT deployment and is the recommended workflow for convolutional and vision tasks. Use PyTorch PTQ only when QAT is required to achieve better accuracy.

[liukang1811]

This is expected. For convolutional models, ONNX PTQ generally performs better for TensorRT deployment and is the recommended workflow for convolutional and vision tasks. Use PyTorch PTQ only when QAT is required to achieve better accuracy.

Thank you very much for your answer. I have browsed many issues about nvidia-modelopt and think you are a very experienced engineer, so I asked you the above questions. But I would like to add a few more questions:

1. I currently want to use this method for QAT, but the PyTorch model needs to be calibrated before QAT. So, does the QAT-quantized model necessarily have worse acceleration performance than the ONNX PTQ-quantized model?

2. Since the core of the nvidia-modelopt tool is PyTorch quantization, will this problem occur with all tools involving PyTorch quantization? (TensorRT will insert additional QuantizeLinear.)

3. What is the core principle behind this problem?

4. Is there a possibility to solve this problem?

[i-riyad]

1. Yes, QAT-quantized INT8/FP8 models typically have worse acceleration performance than ONNX PTQ models when deployed with TensorRT, especially for vision models. However, you should try using Torch-TensorRT after QAT—it may provide performance closer to ONNX PTQ in some cases.
2. No, this performance gap is mainly observed with INT8 and FP8 quantization. For other precisions like FP4, PyTorch-based flows generally do not have this issue.
3. The core issue is that PyTorch-based QAT does not place QuantizeLinear and DequantizeLinear (INT8/FP8 QDQ) nodes in a way that is optimal for TensorRT’s layer fusion and kernel selection. In contrast, ONNX PTQ with modelopt can carefully control QDQ placement to be TensorRT-friendly.
4. This issue can potentially be mitigated by using Torch-TensorRT, which compiles PyTorch QAT models directly and may optimize QDQ placement better than exporting to ONNX. However, it may still not match ONNX PTQ performance for all models. We will publish an example soon showcasing the modelopt QAT → Torch-TensorRT workflow.

[liukang1811]

Thank you for your answer. I have resolved this issue by mimicking the positions of Q/DQ in TensorRT's implicit quantization and adjusting the insertion positions of Q/DQ in my Torch model to closely align with those in the implicit quantization. This solved the problem. The general modifications were as follows:
MYINT8_DEFAULT_CFG = {
"quant_cfg": {
"*weight_quantizer": {"num_bits": 8, "axis": 0},
"*input_quantizer": {"num_bits": 8, "axis": None},

    "*lm_head*": {"enable": False},
    "*block_sparse_moe.gate*": {"enable": False},  # Skip the MOE router
    "*router*": {"enable": False},  # Skip the MOE router
    "*output_layer*": {"enable": False},
    "output.*": {"enable": False},
    **default_disabled_quantizer_cfg,
    "*_blocks_conv.2*_project_conv.1*output_quantizer": {"num_bits": 8, "axis": None},
    "*_blocks_conv.0*_project_conv.1*output_quantizer": {"num_bits": 8, "axis": None},
    "*_blocks_conv.1*float_func.0*output_quantizer": {"num_bits": 8, "axis": None},
    "*_blocks_conv.3*._expand_conv.0*input_quantizer": {"enable": False},
    "*_blocks_conv.4*._expand_conv.0*input_quantizer": {"enable": False},
    "*_blocks_conv.1*._depthwise_conv.0*input_quantizer": {"enable": False},
    "*add_mod_1*.*output_quantizer": {"num_bits": 8, "axis": None},
    "default": {"enable": False},
},
"algorithm": "max",

}

[i-riyad]

Great! Which model is this—EfficientNet or MobileNet? For certain models, QDQ placement can be done just as optimally in PyTorch as in ONNX quantization.

[liukang1811]

Sorry for the late reply, EfficientNet.

[Floriangit12]

Hello,
I'm having the same problem right now.
I understand the problem, but I'm not sure how you managed to place the quantisation layers yourself. My model is a bit complex; it's a 3D occupancy with an implicit MLP head.

@liukang1811, could you explain your approach in a little more detail?
And thank you for asking this question, because I didn't understand where my error was in the QAT phase.

[liukang1811]

Hello, I'm having the same problem right now. I understand the problem, but I'm not sure how you managed to place the quantisation layers yourself. My model is a bit complex; it's a 3D occupancy with an implicit MLP head.

@liukang1811, could you explain your approach in a little more detail? And thank you for asking this question, because I didn't understand where my error was in the QAT phase.

The automatic QDQ insertion in modelopt is relatively simple, but it usually requires some manual tuning. You can’t assume that the QDQ placement done by modelopt will always be optimal. Each model needs to be debugged and adjusted individually, and the solution can vary from case to case.

The recommended approach is to analyze your model piece by piece and manually adjust the positions where QDQ nodes are inserted. When doing this, you should study how TensorRT performs operator fusion in FP16 mode and make sure that your inserted QDQ nodes do not break those operator fusion patterns.

To generate the operator fusion information in FP16 mode, you can use the following command:trtexec --onnx=your.onnx --saveEngine=./your.engine
--exportProfile=your.trt.json
--exportLayerInfo=your.layer.json
--profilingVerbosity=detailed --fp16

[Floriangit12]

Hello, I'm having the same problem right now. I understand the problem, but I'm not sure how you managed to place the quantisation layers yourself. My model is a bit complex; it's a 3D occupancy with an implicit MLP head.
@liukang1811, could you explain your approach in a little more detail? And thank you for asking this question, because I didn't understand where my error was in the QAT phase.

The automatic QDQ insertion in modelopt is relatively simple, but it usually requires some manual tuning. You can’t assume that the QDQ placement done by modelopt will always be optimal. Each model needs to be debugged and adjusted individually, and the solution can vary from case to case.

The recommended approach is to analyze your model piece by piece and manually adjust the positions where QDQ nodes are inserted. When doing this, you should study how TensorRT performs operator fusion in FP16 mode and make sure that your inserted QDQ nodes do not break those operator fusion patterns.

To generate the operator fusion information in FP16 mode, you can use the following command:trtexec --onnx=your.onnx --saveEngine=./your.engine --exportProfile=your.trt.json --exportLayerInfo=your.layer.json --profilingVerbosity=detailed --fp16

Thank you very much.
I'll try that.