Please help, ModelOPT int8 quantized model runs slower than fp16 quantized model.

[Rajjeshwar]

I benchmarked my model with fp16 + int8 engine and fp16 engine and it seems fp16 is faster somehow even though its a timm convnext model with a lot of convolution layers.

Here is a quantized model : https://drive.google.com/file/d/1kFJnHLcFAVFWyrIEvJz-l3TKW0WgsasZ/view?usp=sharing demonstrating my issue.

[cjluo-nv]

Could you share how you evaluate the performance?

[Rajjeshwar]

https://drive.google.com/drive/folders/1m8EhhN5vTVwWEbmOUasCD6ZBAMDkAdfj?usp=sharing

Hey thank you for the response! I provided a notebook with dummy data and model to show how I am quantizing and benchmarking them and a requirements file here.

[Rajjeshwar]

I did some digging and this is how my graph looks :

From what I understand every operation proceeding the conv output (ReduceMean, Sub, Pow etc) seems to be operating in float32. The bias is also in fp32 but I presume since its a fused operation it would not yield any speedup if I manually cast it to int8? My deployment target is a jetson orin NX.

Edit : I should add I have a ViT based model as well with the same issue, I am not sure if the sum operation is causing this problem. Not sure how to add a Q/DQ node there and test. Any help would be really appreciated.

[Rajjeshwar]

Graph before: https://drive.google.com/file/d/1CxkzM3bwF4xGbvmH-lZ6gq2Y6j4DlQw1/view?usp=drive_link

Graph after: https://drive.google.com/file/d/1tApEtZRYEkShXjjH1d9N6TvgZWBJBAD2/view?usp=drive_link

I am now using opset 17 to export, layernorm operation is therefore supported, I also quantized the GELU and GRN operations (not checking performance at the moment) but even then it is quite a bit slower than fp16.

[WhiteTeaDragon]

Has this issue been resolved? I encounter the same problems with performance of a quantized model

[Raj-vivid]

Has this issue been resolved? I encounter the same problems with performance of a quantized model

Nope sadly it didnt. From what I investigated (in my convnext model) this is mainly an issue with how onnx opsets convert from torch and how modelopt or pytorchquantization reads this onnx graph. A lot of operations didnt have specific nodes and get reduced to their basic operations which entails, tensorrt cant just add q/dq nodes correctly. You would have to add one before each add/multiple/dive etc and the more q/dq nodes in your graph the slower the model becomes.

Using the latest opsets on onnx you can get nodes for GeLU, LayerNorm nodes and add q/dq before or after them and you can use pytorch-quantization module of modelopt to manually add q/dq for residual add. However, for GRN layer that convnext uses there is no onnx node so you would have to make one using graph surgeon which I wasn't able to get right (outputs were not matching). Hopefully someone else can solve this. :/

[WhiteTeaDragon]

@Raj-vivid
Thank you for your answer! So, you are saying that adding q/dq operations leads to worse performance? What if I want each operation to run in int8? Shouldn't I add those layers before each operation? The model converted via trtexec with --int8 flag runs much faster than what I get after modelopt -- so, in general, it is possible to quantize the model effectively.

[Raj-vivid]

I think if you want to run each operation in int8 you first want to fuse them into one node/kernel. These operations default to f32 so the more q/dq nodes you add the more number of times the model is converting the f32 <-> int8. You typically want as few q/dq nodes as possible to ensure a whole bunch of computations are done in int8 without converting back. For this purpose tensorrt has its own fusions (conv-gelu or conv-bn-relu etc) where you dont have to explicitly put anything before each layer as long as there is a q/dq following that group of operations it does the whole thing in one kernel. However thats not the case when you have say add-divide-reshape-mul. You could again write your own custom fused kernel for this but I am not too great with that process.

Also, its interesting simply setting trtexec --int8 flag works for you, I end up with errors. Could you tell me what model you are using? Is it convnextv2?

[WhiteTeaDragon]

Sorry to disappoint you, but I am converting Hi-Fi GAN to trt -- it also has convolutions (although 1d) and residual connections, but it does not have a GRN layer. I have not encountered any issues with --int8 flag for my case.

Thank you for the info! I noticed that in my case modelopt adds a q/dq layer before each convolution -- so, it seems that performance issues are not considered. Is there a comprehensive list of operations that can be fused in trt?

[i-riyad]

The issue with applying the default quantization scheme in this model (and a few others) is that TensorRT cannot fuse quantized Conv and Add operations when they use different scaling factors. Addressing this requires graph-level analysis to group certain tensors under shared scales.

As a workaround, you can try one of the following:

• Set DedicatedQDQPair to False to allow shared Q/DQ pairs
• Or use the flag --op_types_to_quantize Conv to limit quantization to convolution layers only (which, in effect, achieves the same outcome)

We’re working on a more robust solution and expect to improve this behavior soon.

[WhiteTeaDragon]

@i-riyad

Should I change the source code to use the first option?

As for the second option, do I understand correctly than only convolutional layers will use int8 then? Wouldn't it mean that a lot of time will be spent on convertation from int8 at the Conv output to fp32 at the Add input? I have tried this option and did not notice any speedup (TRT 10.9, H100, Hi-Fi GAN):

non-quantized onnx:

• trtexec with no additional options: 25 ms
• --fp16: 15 ms
• --int8: 13 ms

onnx quantized from torch:

• --stronglyTyped: 28 ms

onnx quantized from onnx:

• --stronglyTyped: 26 ms

onnx quantized from onnx with --op_types_to_quantize Conv:

• --stronglyTyped: 28 ms

[i-riyad]

Sorry for the confusion.
Here are my benchmark results (median GPU Compute Time) on an H100 for this model:
Baseline (no quantization):
trtexec --onnx=non_quant.onnx --fp16
→ 8.34052 ms

TensorRT implicit quantization (without explicit QDQ):
trtexec --onnx=non_quant.onnx --fp16 --int8 --builderOptimizationLevel=4
→ 7.08472 ms

Default quantization with modelopt led to a performance regression:
python -m modelopt.onnx.quantization --onnx_path=non_quant.onnx --output_path=quant.onnx --quantize_mode=int8
trtexec --onnx=quant.onnx --fp16 --int8 --builderOptimizationLevel=4
→ 9.91626 ms

Quantizing only Conv ops using modelopt significantly improved performance:
python -m modelopt.onnx.quantization --onnx_path=non_quant.onnx --output_path=quant2.onnx --quantize_mode=int8 --op_types_to_quantize Conv
trtexec --onnx=quant2.onnx --fp16 --int8 --builderOptimizationLevel=4 --exportLayerInfo=quant2.engine.graph.json
→ 6.81714 ms

Note:
The --builderOptimizationLevel=4 flag is important for maximizing TensorRT optimization.
⸻

Wouldn’t it mean that a lot of time will be spent on conversion from INT8 at the Conv output to FP32 at the Add input?

Not necessarily.
TensorRT has epilogue and prologue fusions, which can efficiently fuse adjacent layers like Conv, MatMul, and Gemm with operations such as Add, Bias, or Activation.
You can check the actual fusion behavior in the exported engine layer info (quant2.engine.graph.json).

Even if TensorRT cannot fully fuse an Add layer with a preceding Conv, the overall performance can still improve because heavy computations are still performed in low precision (INT8).
⸻

Also, could you share the Hi-Fi GAN ONNX model?
@WhiteTeaDragon

[Rajjeshwar]

Oh hey! Sorry I didn't get time to check the responses here since I am travelling. Thank you for checking it out!

A few questions,

• I tried quantizing just the conv modules but it didn't yield any speed ups for me. Do we need a specific version of tensorrt for this (I am not sure why else it wouldn't work for me)
• Does it need specific hardware? The deployment platform for me is a Jetson orin nx.

Thank you for the response though. I am going to try this out on Monday and get back to you. The performance increase is quite noticeable here.

[i-riyad]

• I benchmarked the model using TensorRT 10.9, and I can confirm that TensorRT version does matter — you will need at least TensorRT 10.x for (near) optimal results.
• I will also ask someone from our team to benchmark the model on Orin, which will definitely yield different numbers compared to the H100 and A6000 GPUs I have been using.

[Rajjeshwar]

Thank you! That would be really helpful!

Also, would quantizing linear layers associated with a model like this (say as a regression head) cause any problems with the conversions? Since I don't think those get fused like the conv layers do?