Int8 mode is slower than fp16

[ye1024]

Hi,
I took out the token embedding layer in Bert and built tensorrt engine to test the inference effect of int8 mode, but found that int8 mode is slower than fp16；
i use nvprof to view the GPU consumption of the two modes, as follows:

fp16:
GPU activities: 99.87% 22.158ms 6 3.6930ms 1.7280us 22.148ms [CUDA memcpy HtoD]
0.06% 13.376us 8 1.6720us 1.6000us 1.9520us [CUDA memset]
0.05% 10.688us 1 10.688us 10.688us 10.688us void cuGatherLayer::gatherGeneric<float, int=32>(void*, cuGatherLayer::StrideArray, cuGatherLayer::gatherGeneric<float, int=32>, void*, int*, void*, cuGatherLayer::ShapeArray, int*, int*, int, cuGatherLayer::ReducedDivisorArray, int, int, int, int, cuGatherLayer::CoefficientData, cuGatherLayer::CoefficientIndices)
0.02% 4.1600us 1 4.1600us 4.1600us 4.1600us [CUDA memcpy DtoH]
0.01% 1.6320us 1 1.6320us 1.6320us 1.6320us [CUDA memcpy DtoD]

int8:
GPU activities: 99.84% 20.210ms 6 3.3683ms 1.6950us 20.201ms [CUDA memcpy HtoD]
0.07% 13.536us 8 1.6920us 1.6000us 1.9840us [CUDA memset]
0.07% 13.311us 1 13.311us 13.311us 13.311us void cuGatherLayer::gatherAxisZeroPartition<float, int=64, int=256>(void*, cuGatherLayer::StrideArray, cuGatherLayer::gatherAxisZeroPartition<float, int=64, int=256>, void*, int*, void*, cuGatherLayer::ShapeArray, int*, int*, int, cuGatherLayer::ReducedDivisorArray, cuGatherLayer::ShapeArray, cuGatherLayer::ShapeArray, int, int, int, int, int, int, nvinfer1::rt::reduced_divisor)
0.02% 3.7120us 1 3.7120us 3.7120us 3.7120us [CUDA memcpy DtoH]
0.01% 1.7280us 1 1.7280us 1.7280us 1.7280us [CUDA memcpy DtoD]

I want to know if there is something wrong with int8 quantization.
Thanks!

TensorRT Version: 6.0.1.5
GPU Type: V100
Nvidia Driver Version: 418.39
CUDA Version: 10.1
Operating System: ubuntu18.04

[ttyio]

Hello @ye1024 , thanks for reporting.
V100 has native INT8 and FP16, but the tensorcore in this chip only support FP16, so we donot always get benefit in INT8. Like this public BERT perf on V100 https://github.com/NVIDIA/TensorRT/tree/master/demo/BERT#inference-performance-nvidia-v100-16gb, FP16 is slight faster.

So I suggest enable both FP16 and INT8 in the builderConfig to allow mix precision for V100. And let TRT to select best kernel for your model. Also try RTX GPUs with INT8 tensorCores if possible.

[ye1024]

Hello @ye1024 , thanks for reporting.
V100 has native INT8 and FP16, but the tensorcore in this chip only support FP16, so we donot always get benefit in INT8. Like this public BERT perf on V100 https://github.com/NVIDIA/TensorRT/tree/master/demo/BERT#inference-performance-nvidia-v100-16gb, FP16 is slight faster.

So I suggest enable both FP16 and INT8 in the builderConfig to allow mix precision for V100. And let TRT to select best kernel for your model. Also try RTX GPUs with INT8 tensorCores if possible.

Thanks for the reply.

I will try to set the mixing precision. In addition, I want to know whether the tensorcore of T4 supports int8?

[ttyio]

@ye1024 Yes, T4 supports int8 tensor core.

[ye1024]

@ye1024 Yes, T4 supports int8 tensor core.

Thank you for your answer. It's very helpful to me.
I want to know if we don't do int8 quantization training and use calibrator to do int8 quantization, will it slow down the inference speed or only lose the accuracy?

[Bonsen]

@ye1024 I also have this question! I create calibrator in python, and then set calibrator in config of C++.
Do u know it? @ttyio

[ttyio]

Hello @ye1024 @Bonsen , sorry I missed this thread,
For PTQ we will quantize all layer outputs by default, and need user manually set mixed precision when they found accuracy drop. We can think PTQ has consider perf only by default.

For QAT, we will only quantize the input of layers that known can be quantized in most cases, so we can think QAT has considered both accuracy and perf.

However it is not a must that PTQ has lower accuracy than QAT in all cases. So my suggestion is that if the find-tuning is not hard, let's try both QAT and PTQ, and choose the better one.

[Bonsen]

THANKS

[twmht]

@ttyio

I also found out QAT is slower than fp16 and PTQ.

so how to force QAT to quantize all layer's output? Why we can get better speed when quantize all layer's output? for example, say that we have the following graph

conv1->relu1->conv2

For QAT we only quantize the input of conv1 and conv2, but PTQ quantize the input of relu1 also (that is from conv1's output)?

[ttyio]

@twmht , for the simple conv->relu->conv pattern, there is no difference between PTQ and QAT, the difference come from some complex pattern, e.g, residual for resnet, see
https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/tutorials/quant_resnet50.html#further-optimization

[twmht]

@ttyio

so if we have conv->relu, and the residual input

(conv->relu) + residual, according to the document you ref, do we need to quantize the output of relu to fuse the elementwise operator? that is, quantize(conv)->quantize(relu) + quantize (residual) ?

Here the fused operator’s output precision must match the residual input precision

what if the precision don't match? what would tensorrt do?

[twmht]

@ttyio
I also found out if my graph activation function is leaky relu.

like conv1->leaky_relu->conv2, tensorrt would say conv2 is missing quantization data.

tensorrt log would print following messages

[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_2 due to missing int8 scales for tensor 427 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_3 due to missing int8 scales for tensor 427 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_6 due to missing int8 scales for tensor 296 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_7 due to missing int8 scales for tensor 430 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_10 due to missing int8 scales for tensor 301 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_11 due to missing int8 scales for tensor 433 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_14 due to missing int8 scales for tensor 436 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_15 due to missing int8 scales for tensor 436 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_18 due to missing int8 scales for tensor 311 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_19 due to missing int8 scales for tensor 439 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_22 due to missing int8 scales for tensor 316 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_23 due to missing int8 scales for tensor 442 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_26 due to missing int8 scales for tensor 445 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_27 due to missing int8 scales for tensor 445 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_30 due to missing int8 scales for tensor 326 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_31 due to missing int8 scales for tensor 448 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_34 due to missing int8 scales for tensor 331 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_35 due to missing int8 scales for tensor 451 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_38 due to missing int8 scales for tensor 454 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_39 due to missing int8 scales for tensor 454 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_42 due to missing int8 scales for tensor 341 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_46 due to missing int8 scales for tensor 346 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_47 due to missing int8 scales for tensor 460 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_50 due to missing int8 scales for tensor 463 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_51 due to missing int8 scales for tensor 463 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_54 due to missing int8 scales for tensor 356 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_55 due to missing int8 scales for tensor 466 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_58 due to missing int8 scales for tensor 361 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_59 due to missing int8 scales for tensor 469 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_62 due to missing int8 scales for tensor 472 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_63 due to missing int8 scales for tensor 472 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_66 due to missing int8 scales for tensor 371 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_67 due to missing int8 scales for tensor 475 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_70 due to missing int8 scales for tensor 376 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_71 due to missing int8 scales for tensor 478 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_74 due to missing int8 scales for tensor 481 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_75 due to missing int8 scales for tensor 481 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_78 due to missing int8 scales for tensor 386 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_79 due to missing int8 scales for tensor 484 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_82 due to missing int8 scales for tensor 391 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_83 due to missing int8 scales for tensor 487 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_86 due to missing int8 scales for tensor 490 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_87 due to missing int8 scales for tensor 490 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_90 due to missing int8 scales for tensor 401 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_94 due to missing int8 scales for tensor 406 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_95 due to missing int8 scales for tensor 496 at input index 0
[TensorRT] WARNING: Rejecting some int8 implementation of layer Conv_98 due to missing int8 scales for tensor 499 at output index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_99 due to missing int8 scales for tensor 499 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_102 due to missing int8 scales for tensor 416 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer LeakyRelu_103 due to missing int8 scales for tensor 502 at input index 0
[TensorRT] VERBOSE: Rejecting some int8 implementation of layer Conv_106 due to missing int8 scales for tensor 421 at input index 0
[TensorRT] VERBOSE: *************** Autotuning Reformat:Float(695040,231680,362,1) -> Float(695040,1,1086,3) ***************

and the tensorrs for feeding to leaky_relu would print the following messages

[TensorRT] WARNING: Missing scale and zero-point for tensor 427, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 296, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 430, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 301, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 433, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 436, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor 311, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
...