Fix: fix float point 16 precision type issue #1853

bowang007 · 2023-04-22T00:52:58Z

Description

For some networks running with FP16, some bugs like

Expected input tensors to have type Half, found type float

are observed.
This PR fixes the bug that output type may not match to the input type
in converters. Some of the converters should make sure that the input
tensor and output tensor have the same date type.

Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes
I have added the relevant labels to my PR in so that relevant reviewers are notified

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

gs-olive · 2023-04-25T01:45:26Z

core/conversion/conversion.cpp

+    if (ctx->enabled_precisions.find(cur_input_type) == ctx->enabled_precisions.end()) {
+      TORCHTRT_THROW_ERROR("Input type is not found in enabled_precisions.");
+    }


This error/check can cause models to erroneously fail, as many models (e.g. transformers) require torch.int32 inputs, but are compiled with torch.float or torch.half precision.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

This PR fixes the bug that output type may not match to the input type in converters. Some of the converters should make sure that the input tensor and output tensor have the same date type.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

bowang007 · 2023-06-07T03:07:00Z

@narendasan What actually is happening is that in Converters the output type for each layer is not correctly inferred.
The updated PR addresses such issues by inferring output types for some converters instead of setting whole TRT engine's output type.
Please take a look, and I think some other converters may need to be updated as well.

narendasan · 2023-06-07T21:39:40Z

core/conversion/converters/impl/activation.cpp

@@ -75,9 +77,10 @@ auto acthardtanh TORCHTRT_UNUSED =
                    new_layer->setBeta(max);

                    new_layer->setName(util::node_info(n).c_str());
+                    new_layer->setOutputType(0, in->getType());


What would cause this to not be the case? would this interfere with TensorRT's mixed precision behavior?

bowang007 · 2023-11-15T01:12:10Z

closing per #2469

facebook-github-bot added the cla signed label Apr 22, 2023

github-actions bot requested a review from peri044 April 22, 2023 00:53

github-actions bot approved these changes Apr 22, 2023

View reviewed changes

bowang007 force-pushed the fix_fp16_bug branch from 150682d to 5dde43c Compare April 22, 2023 00:55

github-actions bot approved these changes Apr 22, 2023

View reviewed changes

gs-olive reviewed Apr 25, 2023

View reviewed changes

gs-olive mentioned this pull request Apr 28, 2023

❓ [Question] Runtimes for timm + TensorRT #1860

Closed

github-actions bot approved these changes Jun 3, 2023

View reviewed changes

bowang007 force-pushed the fix_fp16_bug branch from a906a27 to 9770981 Compare June 3, 2023 00:16

github-actions bot approved these changes Jun 3, 2023

View reviewed changes

bowang007 force-pushed the fix_fp16_bug branch from 9770981 to ef31371 Compare June 3, 2023 00:19

github-actions bot approved these changes Jun 3, 2023

View reviewed changes

github-actions bot added the component: converters Issues re: Specific op converters label Jun 6, 2023

This comment was marked as outdated.

Sign in to view

github-actions bot approved these changes Jun 6, 2023

View reviewed changes

This comment was marked as outdated.

Sign in to view

github-actions bot approved these changes Jun 6, 2023

View reviewed changes

fix: fix converter output type bug

7e17b70

This PR fixes the bug that output type may not match to the input type in converters. Some of the converters should make sure that the input tensor and output tensor have the same date type.

bowang007 force-pushed the fix_fp16_bug branch from fea7ca0 to 7e17b70 Compare June 6, 2023 05:22

github-actions bot approved these changes Jun 6, 2023

View reviewed changes

narendasan reviewed Jun 7, 2023

View reviewed changes

garg-aayush mentioned this pull request Jul 11, 2023

🐛 [Bug] Unable to compile DPT Midas depth estimation model using Torch-TensorRT. #2083

Closed

bowang007 closed this Nov 15, 2023

bowang007 mentioned this pull request Nov 15, 2023

feat: introduce type propagation infrastructure #2469

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: fix float point 16 precision type issue #1853

Fix: fix float point 16 precision type issue #1853

bowang007 commented Apr 22, 2023 •

edited

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Apr 25, 2023 •

edited

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

This comment was marked as outdated.

github-actions bot left a comment

This comment was marked as outdated.

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

bowang007 commented Jun 7, 2023

narendasan Jun 7, 2023

bowang007 commented Nov 15, 2023

Fix: fix float point 16 precision type issue #1853

Fix: fix float point 16 precision type issue #1853

Conversation

bowang007 commented Apr 22, 2023 • edited

Description

Type of change

Checklist:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Apr 25, 2023 • edited

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

github-actions bot left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

bowang007 commented Jun 7, 2023

narendasan Jun 7, 2023

Choose a reason for hiding this comment

bowang007 commented Nov 15, 2023

bowang007 commented Apr 22, 2023 •

edited

gs-olive Apr 25, 2023 •

edited