[Quant] Add fused ConvAdd2d module for onednn backend #91152

leslie-fang-intel · 2022-12-20T03:54:56Z

Stack from ghstack (oldest at bottom):

Summary
Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused ConvAdd2d module for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this module with other quantization backends otherwise an error is thrown.

Test plan

python -m pytest test_quantization.py -k test_conv2d_add

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

[ghstack-poisoned]

pytorch-bot · 2022-12-20T03:55:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91152

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2366a9f:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 83915ff79444b1e26eb6afda3112ef599342e944 Pull Request resolved: pytorch#91152

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 73cd56e761d35b1dab38ebf2429ac54d51f552ef Pull Request resolved: pytorch#91152

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 82712796ce3eb709f9225d52eb0d174291392659 Pull Request resolved: pytorch#91152

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 426bded8b3473fe7277b6fd8e99f9752e0b49e16 Pull Request resolved: pytorch#91152

ghstack-source-id: 2dfaa66d06a073235e5d077d4274ae34c20dad8e Pull Request resolved: pytorch#91152

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `ConvAdd2d` module for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this module with other quantization backends otherwise an error is thrown. **Test plan** ``` python -m pytest test_quantization.py -k test_conv2d_add ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

jerryzh168 · 2023-01-27T07:20:52Z

test/quantization/core/test_quantized_module.py

@@ -274,14 +274,26 @@ def _test_conv_api_impl(
            batch_size, in_channels_per_group, input_feature_map_size,
            out_channels_per_group, groups, kernel_size, X_scale, X_zero_point,
            W_scale, W_zero_point, use_bias, use_channelwise)
+


there are a lot of if/else here in this function, I'm wondering if we can pass them as arguments,
e.g. can we pass
1). inputs to the module as arguments ([[X_q, X2_q]] if (post_op == "add") else [[X_q]])
2). separate the fused_conv_module and the individual conv_module as two arguments, to avoid branches in L296-303
3). any other refactors to remove branches

Thanks for the suggestions. I have removed most of the if/else branches in this function. Please help to take a look again @jerryzh168.

jerryzh168

I think we should remove the branches as much as we can to make the code clearer

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `ConvAdd2d` module for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this module with other quantization backends otherwise an error is thrown. **Test plan** ``` python -m pytest test_quantization.py -k test_conv2d_add ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel · 2023-01-28T06:04:08Z

I think we should remove the branches as much as we can to make the code clearer

Most of the branches has been removed.

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `ConvAdd2d` module for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this module with other quantization backends otherwise an error is thrown. **Test plan** ``` python -m pytest test_quantization.py -k test_conv2d_add ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel · 2023-01-31T01:20:42Z

Hi @jerryzh168, thanks for your review. the previous comments has been fixed. Could you kindly take a look of this PR again?

jerryzh168 · 2023-01-31T01:31:59Z

test/quantization/core/test_quantized_module.py

+            batch_size = 2
+            in_channels_per_group = 2
+            H = 8
+            W = 8
+            out_channels_per_group = 2
+            groups = 3
+            kernel_h = 3
+            kernel_w = 3
+            stride_h = 2
+            stride_w = 2
+            pad_h = 1
+            pad_w = 1
+            dilation = 1
+            # Tests the correctness of the conv2d module.
+            in_channels = in_channels_per_group * groups
+            out_channels = out_channels_per_group * groups
+            input_feature_map_size = (H, W)
+            kernel_size = (kernel_h, kernel_w)
+            stride = (stride_h, stride_w)
+            padding = (pad_h, pad_w)
+            dilation = (dilation, dilation)
+            X_scale = 1.3
+            X_zero_point = 2
+            W_scale = [0.5]
+            W_zero_point = [0] if qengine_is_onednn() else [3]
+            Y_scale = 5.0
+            Y_zero_point = 4
+            qconv_cls = nniq.ConvReLU2d
+            module_name = "QuantizedConvReLU2d"


nit: can you move these outside of the loop?

Thanks for the comments. Done.

jerryzh168 · 2023-01-31T01:32:17Z

test/quantization/core/test_quantized_module.py

+        for use_bias, use_channelwise in options:
+            if torch.backends.quantized.engine == "qnnpack":
+                use_channelwise = False
+            batch_size = 2


same for these

jerryzh168 · 2023-01-31T01:32:26Z

test/quantization/core/test_quantized_module.py

+                [True, False],  # use_channelwise
+            )
+            for pad_mode, use_bias, use_channelwise in options:
+                batch_size = 2


Also finished here.

jerryzh168 · 2023-01-31T01:39:23Z

torch/ao/nn/intrinsic/modules/fused.py

+class ConvAdd2d(_FusedModule):
+    r"""This is a sequential container which calls the Conv2d modules with extra Add.
+    During quantization this will be replaced with the corresponding fused module."""
+    def __init__(self, add, conv):


nit: can we swap the order of conv and add, to be more consistent with the other ops

Good suggestion, changed the order to make it more consistent.

jerryzh168

looks good, had a few comments inline

jerryzh168 · 2023-01-31T01:41:40Z

torch/ao/nn/intrinsic/modules/fused.py

+    def forward(self, x1, x2):
+        return self.add(self[0](x1), x2)


also this feels a bit hacky, i'm wondering how we can extend _FusedModule to support this use case..

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `ConvAdd2d` module for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this module with other quantization backends otherwise an error is thrown. **Test plan** ``` python -m pytest test_quantization.py -k test_conv2d_add ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel · 2023-02-01T01:09:44Z

@pytorchbot merge

pytorchmergebot · 2023-02-01T01:11:21Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[Quant] Add fused ConvAdd2d module for onednn backend

3d313c9

[ghstack-poisoned]

leslie-fang-intel requested review from jerryzh168 and z-a-f as code owners December 20, 2022 03:54

pytorch-bot bot added the release notes: quantization release notes category label Dec 20, 2022

leslie-fang-intel marked this pull request as draft December 20, 2022 03:55

leslie-fang-intel changed the title ~~[Quant] Add fused ConvAdd2d module for onednn backend~~ [WIP] [Quant] Add fused ConvAdd2d module for onednn backend Dec 20, 2022

leslie-fang-intel added ciflow/trunk Trigger trunk jobs on your pull request intel This tag is for PR from Intel labels Dec 20, 2022

leslie-fang-intel requested review from jgong5 and Xia-Weiwen December 20, 2022 03:56

leslie-fang-intel added the open source label Dec 20, 2022

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 20, 2022

[Quant] Add fused ConvAdd2d module for onednn backend

7fedcc2

ghstack-source-id: 83915ff79444b1e26eb6afda3112ef599342e944 Pull Request resolved: pytorch#91152

Update on "[WIP] [Quant] Add fused ConvAdd2d module for onednn backend"

ae00032

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 20, 2022

[Quant] Add fused ConvAdd2d module for onednn backend

3c35de3

ghstack-source-id: 73cd56e761d35b1dab38ebf2429ac54d51f552ef Pull Request resolved: pytorch#91152

leslie-fang-intel added 2 commits December 20, 2022 15:11

Update on "[WIP] [Quant] Add fused ConvAdd2d module for onednn backend"

5643dff

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

Update on "[WIP] [Quant] Add fused ConvAdd2d module for onednn backend"

765df56

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 20, 2022

[Quant] Add fused ConvAdd2d module for onednn backend

87dc5e1

ghstack-source-id: 82712796ce3eb709f9225d52eb0d174291392659 Pull Request resolved: pytorch#91152

leslie-fang-intel added 2 commits December 21, 2022 13:28

Update on "[WIP] [Quant] Add fused ConvAdd2d module for onednn backend"

e08f924

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

Update on "[WIP] [Quant] Add fused ConvAdd2d module for onednn backend"

2811925

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

leslie-fang-intel changed the title ~~[WIP] [Quant] Add fused ConvAdd2d module for onednn backend~~ [Quant] Add fused ConvAdd2d module for onednn backend Dec 21, 2022

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 21, 2022

[Quant] Add fused ConvAdd2d module for onednn backend

32c3388

ghstack-source-id: 426bded8b3473fe7277b6fd8e99f9752e0b49e16 Pull Request resolved: pytorch#91152

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Jan 13, 2023

[Quant] Add fused ConvAdd2d module for onednn backend

34dc8bb

ghstack-source-id: 2dfaa66d06a073235e5d077d4274ae34c20dad8e Pull Request resolved: pytorch#91152

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Jan 26, 2023

[Quant] Add fused ConvAdd2d module for onednn backend

69b9546

ghstack-source-id: 2dfaa66d06a073235e5d077d4274ae34c20dad8e Pull Request resolved: pytorch#91152

leslie-fang-intel added 3 commits January 26, 2023 10:14

jerryzh168 reviewed Jan 27, 2023

View reviewed changes

jerryzh168 requested changes Jan 27, 2023

View reviewed changes

leslie-fang-intel added 4 commits January 28, 2023 11:02

leslie-fang-intel requested review from jerryzh168 and removed request for z-a-f January 28, 2023 06:04

leslie-fang-intel added 2 commits January 28, 2023 14:32

jerryzh168 reviewed Jan 31, 2023

View reviewed changes

jerryzh168 approved these changes Jan 31, 2023

View reviewed changes

jerryzh168 reviewed Jan 31, 2023

View reviewed changes

pytorchmergebot added the Merged label Feb 1, 2023

pytorchmergebot closed this in 53c3555 Feb 1, 2023

facebook-github-bot deleted the gh/leslie-fang-intel/14/head branch June 8, 2023 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] Add fused ConvAdd2d module for onednn backend #91152

[Quant] Add fused ConvAdd2d module for onednn backend #91152

leslie-fang-intel commented Dec 20, 2022 •

edited

pytorch-bot bot commented Dec 20, 2022 •

edited

jerryzh168 Jan 27, 2023

leslie-fang-intel Jan 28, 2023

jerryzh168 left a comment

leslie-fang-intel commented Jan 28, 2023

leslie-fang-intel commented Jan 31, 2023

jerryzh168 Jan 31, 2023

leslie-fang-intel Jan 31, 2023

jerryzh168 Jan 31, 2023

leslie-fang-intel Jan 31, 2023

jerryzh168 Jan 31, 2023

leslie-fang-intel Jan 31, 2023

jerryzh168 Jan 31, 2023

leslie-fang-intel Jan 31, 2023

jerryzh168 left a comment

jerryzh168 Jan 31, 2023

leslie-fang-intel commented Feb 1, 2023

pytorchmergebot commented Feb 1, 2023

[Quant] Add fused ConvAdd2d module for onednn backend #91152

[Quant] Add fused ConvAdd2d module for onednn backend #91152

Conversation

leslie-fang-intel commented Dec 20, 2022 • edited

pytorch-bot bot commented Dec 20, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91152

✅ No Failures

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryzh168 left a comment

Choose a reason for hiding this comment

leslie-fang-intel commented Jan 28, 2023

leslie-fang-intel commented Jan 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryzh168 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leslie-fang-intel commented Feb 1, 2023

pytorchmergebot commented Feb 1, 2023

Merge started

leslie-fang-intel commented Dec 20, 2022 •

edited

pytorch-bot bot commented Dec 20, 2022 •

edited