Can the outputs of pytorch and tensorrt be accurately aligned ？

[lzcchl]

In fact, I am currently working on a project to verify whether the outputs of pytorch and tensorrt can be accurately aligned.

I have tested both cases of conv and conv+bn, and the netron visualizations of the two models are shown in the following figure.

After testing, under model conv, the results of pytorch and tensorrt are completely consistent.

However, under model conv+bn, there is a difference of approximately 1e-6 in accuracy between the results of pytorch and tensorrt.

I suspect that Tensorrt did some optimization work, such as operator fusion. When building the Tensorrt engine, I added – builderOptimizationLevel=0, I think operator fusion may not be performed at this optimization level, but there is still a slight difference in accuracy between the final results of pytorch and tensorrt.

Also, I try "--builderOptimizationLevel=0 --noTF32 --precisionConstraints=obey --layerPrecisions=:fp32 --layerOutputTypes=:fp32" when build trt engine, but the difference still exist.

So, may I ask what is the internal reason for the difference in results, and is there any way to make this difference disappear ?

Environment
TensorRT Version: 8.6.1
GPU Type: GTX1060
Nvidia Driver Version: 545.29.06
CUDA Version: 12.1
CUDNN Version: 8.9.7
Operating System + Version: ubuntu20.04
Python Version (if applicable): python3.8
TensorFlow Version (if applicable): NO
PyTorch Version (if applicable): 2.1.2+cu121
Baremetal or Container (if container which image + tag): NO
Triton infer server : docker image r23.12

Relevant Files
my onnx(from pytorch) and trt models are here:
test_model.zip

Steps To Reproduce
The following is a screenshot of the source code for torchvision.resnet. I added a return to end the inference earlier.

After entering a data for infer, you can also obtain that the result of “resnet50-part-conv” is the same, but there are differences in the result of “resnet50-part-cb”.

this is my code for consult, I have test torch.backends.cudnn.enabled / benchmark / deterministic, trt infer I use triton.

client-http.zip

[zerollzeng]

We don't guarantee bit-wise accuracy to other framework, it's impossible due to floating point accuracy.

[zerollzeng]

Layer fusion/ kernel optimization also affect the final result too.

[lzcchl]

Does trtexec provide optional parameters to turn off layer fusion and kernel optimization？

[lix19937]

@lzcchl You can turn off layer fusion and kernel optimization, when you modify all nodes of onnx (in other words, add output for every node except the real output nodes),and use --noTF32 , otherwise can not.

[zerollzeng]

Those little numerical error normally won't affect the final mAP.

[lzcchl]

@lix19937 @zerollzeng thanks for reply and good idea. I had add output for conv and BN, the onnx model just like below, I believe it is a model which turn off layer fusion, then convert onnx to trt model by bash as below.

Finally, the output of conv is accurately aligned between pytorch and tensorrt, the output of BN is almost aligned but a little difference as I show as below.

I have no idea about that, I guess there may be slight differences in the implementation of the BN function, resulting in slight differences in the results. It is hard on my future works.

[lix19937]

slight differences are normal.