Quant tensorrt engine don't achieve advantage in inference speed over fp16 on A100

[bigmover]

I replace the unet.trt10.0.1.6.post12.dev1.engine built from Build the TRT engine for the INT8 Quantized ONNX UNet to engine/unet.trt10.0.1.plan . AND rerun the textdemo(python demo_txt2img.py "enchanted winter forest, soft diffuse light on a snow-filled day, serene nature scene, the forest is illuminated by the snow" --negative-prompt "normal quality, low quality, worst quality, low res, blurry, nsfw, nude" --scheduler Euler --denoising-steps 30 --seed 2946901).

Is there something I missed or what mistake I made? Any apply will be appreciated.

fp16

int8

[jingyu-ml]

Hi @bigmover
Could you please share your ONNX model (both FP16 and INT8)? Additionally, you can verify the latency benchmarks after building your engine; the latency information will be displayed at the end.

Please run the following command on your FP16 ONNX models (remove the --int8 flag) and on your INT8 ONNX model. Then, share the log with us. Thank you!

trtexec --onnx=./unet.onnx --shapes=sample:2x4x128x128,timestep:1,encoder_hidden_states:2x77x2048,text_embeds:2x1280,time_ids:2x6 --fp16 --int8 --builderOptimizationLevel=4 --saveEngine=unetxl.{version}.plan

We've tested it internally and it should work. Once you share your ONNX model and the trtexec log, we can work on pinpointing the problem.

[bigmover]

Hi @bigmover Could you please share your ONNX model (both FP16 and INT8)? Additionally, you can verify the latency benchmarks after building your engine; the latency information will be displayed at the end.

Please run the following command on your FP16 ONNX models (remove the --int8 flag) and on your INT8 ONNX model. Then, share the log with us. Thank you!

trtexec --onnx=./unet.onnx --shapes=sample:2x4x128x128,timestep:1,encoder_hidden_states:2x77x2048,text_embeds:2x1280,time_ids:2x6 --fp16 --int8 --builderOptimizationLevel=4 --saveEngine=unetxl.{version}.plan

We've tested it internally and it should work. Once you share your ONNX model and the trtexec log, we can work on pinpointing the problem.

Hi @jingyu-ml !
The fp16 onnx is too large to share. But my model for quantize is the Standard Model from Huggingface(SD 1.5)(fp16 pytorch model).

The int 8 onnx after calibration
unet.onnx.zip

The output int8 log of

TensorRT-10.0.1.6/targets/x86_64-linux-gnu/bin/trtexec --onnx=./sd-1.5-onnx-3/unet.onnx --shapes=sample:2x4x64x64,timestep:1,encoder_hidden_states:2x77x768 --fp16 --builderOptimizationLevel=4 --saveEngine=unet.trt10.0.1.6.post12.dev1_test.engine

trtexec.log

The output fp16 log of

../../TensorRT-10.0.1.6/targets/x86_64-linux-gnu/bin/trtexec --onnx=../../TensorRT/demo/Diffusion/onnx/unet/model.onnx --shapes=sample:2x4x64x64,timestep:1,encoder_hidden_states:2x77x768 --fp
16 --builderOptimizationLevel=4 --saveEngine=unet.trt10.0.1.6.post12.dev1_test.engine > trtexec_fp16.log

trtexec_fp16.log

[jingyu-ml]

Hi @bigmover,

You only provided the ONNX graph, so it's not executable. However, I just tried creating the ONNX model myself. I see you're using an A100 for SD 1.5. The A100 is a very powerful GPU, but running SD 1.5 may not fully utilize its tensor cores, resulting in less optimal speedups when comparing FP16 & INT8. For a single image, the speedup is only about 1.10x.

If you switch to GPUs like the L40s or Ada RTX 6000, you'll notice that for SD 1.5 in FP16, these GPUs perform faster than the A100, due to their VRAM and Tensor Core diff etc. However, for larger models like SDXL, the A100 is much, much faster. In a word, A100 is more suitable for bigger models and bigger batch size.

You might want to consider using less powerful GPUs or increasing the batch size to maximize tensor core utilization, vram and communication bandwidth, etc.

And I just tested this on an A100 and it works well, I can get 1.1+ speedup on the scale of 2 images for SD 1.5, and 1.2-1.3 speedup for A100 on SDXL, 2 images. bigger batch size means bigger utilization and more speedup, you can even try 8 batch size, 16 batch size or even bigger on A100/H100. Let me share each step I took so you can replicate the process:

2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          On  |   00000000:41:00.0 Off |                    0 |
| N/A   25C    P0             41W /  300W |       0MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Tested on A100-PCIE-80GB with CUDA 12.4.

I downloaded the latest version of TensorRT from this location:

TensorRT 10.0 GA for Linux x86_64 and CUDA 12.0 to 12.4 TAR Package

Please verify your system and CUDA version, and download the corresponding TensorRT TAR from here:

https://developer.nvidia.com/tensorrt/download/10x

Run

docker run --rm -it --gpus all nvcr.io/nvidia/pytorch:23.12-py3 /bin/bas

wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.1/tars/TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git

tar -xvzf TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

cd TensorRT-10.0.1.6

export LD_LIBRARY_PATH=$(pwd)/lib:$LD_LIBRARY_PATH

export PATH=$(pwd)/bin:$PATH

cd python

pip install tensorrt-10.0.1-cp<python_version>-*-linux_x86_64.whl

cd ../onnx_graphsurgeon

pip install onnx_graphsurgeon-0.5.0-py2.py3-none-any.whl

cd ../../

git clone https://github.com/NVIDIA/TensorRT.git

cd TensorRT-Model-Optimizer/

pip install "nvidia-modelopt[all]~=0.11.0" --extra-index-url https://pypi.nvidia.com

cd diffusers/

pip install -r requirements.txt

Let's first focus on the performance instead of the image, so we can set the calibration parameters to random numbers.

For SDXL, you can try:

python quantize.py \
  --model stabilityai/stable-diffusion-xl-base-1.0 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method min-mean \
  --percentile 1.0 --alpha 0.8 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sdxl.test

python run_export.py --model stabilityai/stable-diffusion-xl-base-1.0 --quantized-ckpt ./unet.state_dict.sdxl.test.pt --format int8 --quant-level 3.0 --onnx-dir sdxl.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sdxl.test.onnx/unet.onnx --output-onnx ./sdxl.test.onnx/unet.surged.onnx

trtexec --onnx=./sdxl.test.onnx/unet.surged.onnx --shapes=sample:4x4x128x128,timestep:1,encoder_hidden_states:4x77x2048,text_embeds:4x1280,time_ids:4x6 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unetxl.int8.plan > sdxl.int8.log

For SD1.5, you can try

python quantize.py \
  --model runwayml/stable-diffusion-v1-5 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method global_min --alpha 0.7 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sd1.5.test

python run_export.py --model runwayml/stable-diffusion-v1-5 --quantized-ckpt ./unet.state_dict.sd1.5.test.pt --format int8 --quant-level 3.0 --onnx-dir sd1.5.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sd1.5.test.onnx/unet.onnx --output-onnx ./sd1.5.test.onnx/unet.surged.onnx

trtexec --onnx=./sd1.5.test.onnx/unet.surged.onnx --shapes=sample:4x4x64x64,timestep:1,encoder_hidden_states:4x77x768 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unet.int8.plan > sd.int8.log

[jingyu-ml]

@bigmover
Also please run this on both your INT8 and FP16 models and share the profiling file with us, and we will involve the performance team.

nsys profile -t cuda,nvtx,cublas,cublas-verbose,cusparse,cusparse-verbose,cudnn --force-overwrite=true --output=int8.profiling trtexec --loadEngine=int8.plan --avgRuns=10

[bigmover]

Hi @bigmover,

You only provided the ONNX graph, so it's not executable. However, I just tried creating the ONNX model myself. I see you're using an A100 for SD 1.5. The A100 is a very powerful GPU, but running SD 1.5 may not fully utilize its tensor cores, resulting in less optimal speedups when comparing FP16 & INT8. For a single image, the speedup is only about 1.10x.

If you switch to GPUs like the L40s or Ada RTX 6000, you'll notice that for SD 1.5 in FP16, these GPUs perform faster than the A100, due to their VRAM and Tensor Core diff etc. However, for larger models like SDXL, the A100 is much, much faster. In a word, A100 is more suitable for bigger models and bigger batch size.

You might want to consider using less powerful GPUs or increasing the batch size to maximize tensor core utilization, vram and communication bandwidth, etc.

And I just tested this on an A100 and it works well, I can get 1.1+ speedup on the scale of 2 images for SD 1.5, and 1.2-1.3 speedup for A100 on SDXL, 2 images. bigger batch size means bigger utilization and more speedup, you can even try 8 batch size, 16 batch size or even bigger on A100/H100. Let me share each step I took so you can replicate the process:

2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          On  |   00000000:41:00.0 Off |                    0 |
| N/A   25C    P0             41W /  300W |       0MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Tested on A100-PCIE-80GB with CUDA 12.4.

I downloaded the latest version of TensorRT from this location:

TensorRT 10.0 GA for Linux x86_64 and CUDA 12.0 to 12.4 TAR Package

Please verify your system and CUDA version, and download the corresponding TensorRT TAR from here:

https://developer.nvidia.com/tensorrt/download/10x

Run

docker run --rm -it --gpus all nvcr.io/nvidia/pytorch:23.12-py3 /bin/bas

wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.1/tars/TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git

tar -xvzf TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

cd TensorRT-10.0.1.6

export LD_LIBRARY_PATH=$(pwd)/lib:$LD_LIBRARY_PATH

export PATH=$(pwd)/bin:$PATH

cd python

pip install tensorrt-10.0.1-cp<python_version>-*-linux_x86_64.whl

cd ../onnx_graphsurgeon

pip install onnx_graphsurgeon-0.5.0-py2.py3-none-any.whl

cd ../../

git clone https://github.com/NVIDIA/TensorRT.git

cd TensorRT-Model-Optimizer/

pip install "nvidia-modelopt[all]~=0.11.0" --extra-index-url https://pypi.nvidia.com

cd diffusers/

pip install -r requirements.txt

Let's first focus on the performance instead of the image, so we can set the calibration parameters to random numbers.

For SDXL, you can try:

python quantize.py \
  --model stabilityai/stable-diffusion-xl-base-1.0 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method min-mean \
  --percentile 1.0 --alpha 0.8 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sdxl.test

python run_export.py --model stabilityai/stable-diffusion-xl-base-1.0 --quantized-ckpt ./unet.state_dict.sdxl.test.pt --format int8 --quant-level 3.0 --onnx-dir sdxl.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sdxl.test.onnx/unet.onnx --output-onnx ./sdxl.test.onnx/unet.surged.onnx

trtexec --onnx=./sdxl.test.onnx/unet.surged.onnx --shapes=sample:4x4x128x128,timestep:1,encoder_hidden_states:4x77x2048,text_embeds:4x1280,time_ids:4x6 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unetxl.int8.plan > sdxl.int8.log

For SD1.5, you can try

python quantize.py \
  --model runwayml/stable-diffusion-v1-5 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method global_min --alpha 0.7 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sd1.5.test

python run_export.py --model runwayml/stable-diffusion-v1-5 --quantized-ckpt ./unet.state_dict.sd1.5.test.pt --format int8 --quant-level 3.0 --onnx-dir sd1.5.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sd1.5.test.onnx/unet.onnx --output-onnx ./sd1.5.test.onnx/unet.surged.onnx

trtexec --onnx=./sd1.5.test.onnx/unet.surged.onnx --shapes=sample:4x4x64x64,timestep:1,encoder_hidden_states:4x77x768 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unet.int8.plan > sd.int8.log

HI @jingyu-ml Thank you very much for your prompt response!
I test again with batch 2 and gained ~15% up in inference speed on A100 with SD 1.5! But actually there is hardly any acceleration effect when I switch to batch=1. However I gained ~20% over inference speed on A10 which SD 1.5 of batch 1.
So as you said that the A100 has sufficient computation, int8 offers no advantage over fp16. I saw int8 achieve Nearly 2x Faster(about 1.72x) on NVIDIA RTX 6000 Ada GPU https://developer.nvidia.com/blog/tensorrt-accelerates-stable-diffusion-nearly-2x-faster-with-8-bit-post-training-quantization/. So the performance data on the A100 GPU mentioned above(hardly any effect with batch 1 on A100) is acceptable ?

Besides dynamic engine is supported when quant?

[jingyu-ml]

HI @jingyu-ml Thank you very much for your prompt response! I test again with batch 2 and gained ~15% up in inference speed on A100 with SD 1.5! But actually there is hardly any acceleration effect when I switch to batch=1. However I gained ~20% over inference speed on A10 which SD 1.5 of batch 1. So as you said that the A100 has sufficient computation, int8 offers no advantage over fp16. I saw int8 achieve Nearly 2x Faster(about 1.72x) on NVIDIA RTX 6000 Ada GPU https://developer.nvidia.com/blog/tensorrt-accelerates-stable-diffusion-nearly-2x-faster-with-8-bit-post-training-quantization/. So the performance data on the A100 GPU mentioned above(hardly any effect with batch 1 on A100) is acceptable ?

@bigmover
Yes, I would say so. For SD1.5 BS=1 (one image) our internal benchmarks can have 1.10x speedup on A100 80G. Since the A100 is not deigned for such small models like SD 1.5. If you still want to run small models on the A100, as we discussed, try using the maximum batch size that doesn't cause an OOTM issue. This might help you achieve 100% GPU utilization for FP16 and better speedup using INT8.

Besides dynamic engine is supported when quant?

I'm not quite sure I understood your question. Were you referring to dynamic quantization, or something else?

[bigmover]

@jingyu-ml

I'm not quite sure I understood your question. Were you referring to dynamic quantization, or something else?

I mean whether I can calibrated pt--> build dynamic onnx -> build dynamic engine?

For SD1.5 BS=1 (one image) our internal benchmarks can have 1.10x speedup on A100 80G

It's tested on TRT 10 or TRT 9.3.? I can gain 10% with BS=1 only in quant mode =2. The int8 engine use input(fp16-> int8) * weight (int8) --> out(int8 ->fp16) to softmax, right? How many times faster is int8 * int8 compared to that speed of tensor core of fp16?

[jingyu-ml]

I mean whether I can calibrated pt--> build dynamic onnx -> build dynamic engine?

What does "dynamic" mean in this context? Are you referring to dynamic batch size?

It's tested on TRT 10 or TRT 9.3.? I can gain 10% with BS=1 only in quant mode =2. The int8 engine use input(fp16-> int8) * weight (int8) --> out(int8 ->fp16) to softmax, right? How many times faster is int8 * int8 compared to that speed of tensor core of fp16?

It has been tested on both. The int8 engine used here only executes Gemm and Conv operations in int8, while other parts remain in fp16. The int8 * int8 Gemm and Conv operations are 1.9 times faster than those in fp16.

[cjluo-nv]

@bigmover has @jingyu-ml resolved this issue?

[bigmover]

@bigmover has @jingyu-ml resolved this issue?

I understand the issue at hand. And I will close the issue. Thanks for your support!