FP16 output wrong of TensorRT 8.6.1 when running GFPGAN on GPU 3070

[ArtemisZGL]

Description

I tried to convert GFPGAN onnx model to tensorrt. When using FP32, the output is correct. But when using FP16, the output is totally wrong. And I saw some warning may affect the accuracy.

[08/23/2023-11:38:45] [TRT] [W] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[08/23/2023-11:38:45] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[08/23/2023-11:38:45] [TRT] [W] Check verbose logs for the list of affected weights.
[08/23/2023-11:38:45]

Thus, I set thoese layer weights to FP32 explicitly. Most of those weights can work, but still some layer weights not working lbecause of ConstShuffleFusion. Thus, the FP16 engine still get the wrong output. Actually, I dont sure this is the real reason.

Running: ConstShuffleFusion on onnx::Add_738
ConstShuffleFusion: Fusing onnx::Add_738 with (Unnamed Layer* 602) [Shuffle]

Environment

TensorRT Version: 8.6.1

NVIDIA GPU: 3070

NVIDIA Driver Version: 525.125.06

CUDA Version: 11.3

CUDNN Version: 8

Operating System: ubuntu 20.04

Relevant Files

Model link:
https://drive.google.com/file/d/1a80j94FVcHjVBkVQ6jvdRucjvTFi3JYy/view?usp=sharing

Log before Set layer FP32:
https://drive.google.com/file/d/1qXPLsxs2LNJO0CsnRNEOweY4-My4bdfd/view?usp=sharing

Log After Set layer FP32:
https://drive.google.com/file/d/1csq6NyNuTcE7vhAkiTuH9F-SCHsqQ1k9/view?usp=sharing

Steps To Reproduce

Commands or scripts: ./trtexec --onnx=GFPGANv1.3.onnx --saveEngine=GFPGANv1.3.trt --workspace=10000 --fp16 --verbose

Have you tried the latest release?: yes

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): yes

[ArtemisZGL]

#3224 It seems similar with this issue.

[YouSenRong]

Yes, from your description and the log, I think it is the same problem.

Did you set these layers to fp32 explicitly but the layers' precision is still fp16?
In TensorRT 8.4.3, the diff is less between fp32 and fp16, about 1% relative diff.
However, in TensorRT 8.6.1, the diff is about 10%, which can be think as a wrong result.

[ArtemisZGL]

Yes, from your description and the log, I think it is the same problem. Did you set these layers to fp32 explicitly but the layers' precision is still fp16? In TensorRT 8.4.3, the diff is less between fp32 and fp16, about 1% relative diff. However, in TensorRT 8.6.1, the diff is about 10%, which can be think as a wrong result.

Yes, the origin log shows about 100 weights affected, so I set all these layer to fp32 explicitly. But I still got 15 weights affected, and I found they all have ConstShuffleFusion operation.

[zerollzeng]

@ArtemisZGL Hi the "Log After Set layer FP32" is incomplete, I want to know the full command to reproduce the "set precision FP32 not work issue". Could you please provide the command? Thanks!

[ArtemisZGL]

@zerollzeng Sure, I use this code to set layers to fp32 explicitly with command as below.

python onnx2trt.py -o GFPGANv1.3.onnx -e GFPGANv1.3.trt -p mixed --fp32_node_file fp32_nodes.txt 

[zerollzeng]

I think the problem with you code is you forgot to set the precision constrain to obey. see https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Core/BuilderConfig.html#tensorrt.BuilderFlag

Could you please set it and try again? Thanks!

[ArtemisZGL]

I think "strict type" is equivalent to "prefer precision constrain"? @zerollzeng

[zerollzeng]

Try set OBEY_PRECISION_CONSTRAINTS

[zerollzeng]

OBEY_PRECISION_CONSTRAINTS : Require that layers execute in specified precisions. Build fails otherwise.

PREFER_PRECISION_CONSTRAINTS : Prefer that layers execute in specified precisions. Fall back (with warning) to another precision if build would otherwise fail.

[ArtemisZGL]

@zerollzeng I change the code as below but still work.

# self.config.set_flag(trt.BuilderFlag.STRICT_TYPES)
self.config.set_flag(trt.BuilderFlag.OBEY_PRECISION_CONSTRAINTS)

The log still shows warning the same as before, also the result is not correct.

[08/31/2023-10:16:00] [TRT] [V]   List of affected weights: onnx::Add_1032 + (Unnamed Layer* 903) [Shuffle], onnx::Add_1089 + (Unnamed Layer* 959) [Shuffle], onnx::Add_1149 + (Unnamed Layer* 1021) [Shuffle], onnx::Add_1207 + (Unnamed Layer* 1077) [Shuffle], onnx::Add_1247 + (Unnamed Layer* 1121) [Shuffle], onnx::Add_1305 + (Unnamed Layer* 1177) [Shuffle], onnx::Add_1345 + (Unnamed Layer* 1221) [Shuffle], onnx::Add_567 + (Unnamed Layer* 432) [Shuffle], onnx::Add_621 + (Unnamed Layer* 482) [Shuffle], onnx::Add_681 + (Unnamed Layer* 544) [Shuffle], onnx::Add_738 + (Unnamed Layer* 602) [Shuffle], onnx::Add_798 + (Unnamed Layer* 664) [Shuffle], onnx::Add_855 + (Unnamed Layer* 722) [Shuffle], onnx::Add_915 + (Unnamed Layer* 784) [Shuffle], onnx::Add_972 + (Unnamed Layer* 842) [Shuffle]

[zerollzeng]

I checked Add_1032, actually it's not a layer but a constant tensors.

The reshape layers in this image looks like unnecessary and I think it break TRT fusion, could you please try disable it in your code?

[ArtemisZGL]

@zerollzeng sorry for late reply, I will have a try as soon as I can.