Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TensorRT 7 Resize produce inconsistent result with PyTorch Interpolate #457

Closed
xpngzhng opened this issue May 9, 2020 · 5 comments
Closed
Labels
bug Something isn't working repro requested Request more information about reproduction of issue triaged Issue has been triaged by maintainers

Comments

@xpngzhng
Copy link

xpngzhng commented May 9, 2020

I have an HRNet semantic segmentation model trained with Pytorch, using interpolate bilinear align_corners=True
Then I convert it for C++ inference with align_corners=True and align_corners=False, I find that the output of the two mode are almost the same.

Using PyTorch1.5, onnx build from source (almost the latest version, showed 1.7.0), set interpolate with align_corners=False, export onnx model with opset version 10.

Then I use TensorRT 7.0.0.11 for inference in Windows. I find the output is quite different from those of PyTorch, the main problem is the result is almost a shifted and smaller version than the PyTorch version.

Is writing a plugin the only way to solve this problem?

Here are the result
torch align
torch_align_mask
torch no align
torch_mask
torch diff
torch_diff
trt no align
mask
trt torch diff
torch_trt_no_align_diff

@xpngzhng xpngzhng changed the title TensorRT 7 Resize produce inconsistent result with PyTorch TensorRT 7 Resize produce inconsistent result with PyTorch Interpolate May 9, 2020
@OtisZZ
Copy link

OtisZZ commented Jul 21, 2020

I met the same problem.
I am using trt7.1 and inference a model, the result is some pixels movement difference from onnx model inference. And the onnx inference is good to align with the tensorflow inference results.
Is there any settings to make upsample functions the same?

@sixcheng
Copy link

I have the same problem.
Have you solved it?

@donrax
Copy link

donrax commented Sep 30, 2020

The issue is connected to the interpolate (resize) layer in Pytorch. Also, take a look at #361 (comment)

You can try and solve it by using the following:
x = F.interpolate(x, size=(H, W), mode='bilinear', align_corners=True)

Pytorch 1.5.1
Tensorrt 7.1.3 (nvidia doc)
Opset 11

@kevinch-nv
Copy link
Collaborator

@xpngzhng thanks for the report. Can you clarify a few things to help us narrow down the issue?

  • Is this reproducible on the latest TRT version? We've made some updates on what resize modes of ONNX we actually support.
  • Is this reproducible on ONNX opset >= 11? ONNX has increased their range of resize modes since opset 11, and that may the cause of the discrepancy seen between Pytorch and ONNX.

@kevinch-nv kevinch-nv added repro requested Request more information about reproduction of issue triaged Issue has been triaged by maintainers bug Something isn't working labels Dec 28, 2020
@kevinch-nv
Copy link
Collaborator

Closing this issue due to inactivity - please open a new issue if the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working repro requested Request more information about reproduction of issue triaged Issue has been triaged by maintainers
Projects
None yet
Development

No branches or pull requests

5 participants