[Quant] Add fused conv2d_add op for onednn backend #90262

leslie-fang-intel · 2022-12-06T06:25:44Z

Stack from ghstack (oldest at bottom):

Summary
Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused conv2d_add op for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this op with other quantization backends otherwise an error is thrown.

Test Plan

python -m pytest test_quantization.py::TestQuantizedConv

cc @VitalyFedyunin @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @gujinghui @PenghuiCheng @jianyuh @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen

[ghstack-poisoned]

pytorch-bot · 2022-12-06T06:25:47Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90262

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8baef3e:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: ca56717ef6b01387b769bd7b679b3e82d5d696b4 Pull Request resolved: #90262

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

ghstack-source-id: a7f5fb324870a2dbdecb84ed9ee76446f6450002 Pull Request resolved: #90262

ghstack-source-id: a7f5fb324870a2dbdecb84ed9ee76446f6450002 Pull Request resolved: pytorch#90262

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

ghstack-source-id: 324e97d3dcda37d6abad83af61453aaf0e9d209e Pull Request resolved: #90262

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `conv2d_add` op for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this op with other quantization backends otherwise an error is thrown. **Test Plan** ``` python -m pytest test_quantization.py::TestQuantizedConv ``` cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

ghstack-source-id: 2b22fc9ffcf1350eac41ce219dfe6d3110abd668 Pull Request resolved: pytorch#90262

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `conv2d_add` op for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this op with other quantization backends otherwise an error is thrown. **Test Plan** ``` python -m pytest test_quantization.py::TestQuantizedConv ``` cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

jerryzh168 · 2023-01-09T17:56:41Z

test/quantization/core/test_quantized_op.py

@@ -4687,7 +4709,7 @@ def _test_qconv_impl(
           Y_scale=st.floats(4.2, 5.6),
           Y_zero_point=st.integers(0, 4),
           use_bias=st.booleans(),
-           use_relu=st.booleans(),
+           post_op=st.sampled_from(["none", "relu"]),


can be a separate PR, but might make sense to split the conv and conv_relu test as well

Thanks for the suggestions, I have split the conv and conv_relu test and so as the other similar test cases.

jerryzh168 · 2023-01-09T17:56:48Z

test/quantization/core/test_quantized_op.py

@@ -4780,7 +4886,7 @@ def test_qconv2d(
           Y_scale=st.floats(4.2, 5.6),
           Y_zero_point=st.sampled_from([0]),
           use_bias=st.booleans(),
-           use_relu=st.booleans(),
+           post_op=st.sampled_from(["none", "relu"]),


Thanks for the suggestions, split them into separate tests.

jerryzh168 · 2023-01-09T18:08:35Z

test/quantization/core/test_quantized_op.py

+            if post_op == "add":
+                qconv = torch.ops.quantized.conv2d_add


if this is only "add" we can remove the post_op argument and also this check

Thanks for the suggestions, I have removed the post_op argument and the check. In next PR, I will put conv2d_add_relu into a separate test.

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `conv2d_add` op for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this op with other quantization backends otherwise an error is thrown. **Test Plan** ``` python -m pytest test_quantization.py::TestQuantizedConv ``` cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

ghstack-source-id: bb2010eccb3737ed8d8706fa43c7c24982a64b72 Pull Request resolved: pytorch#90262

**Summary** Post op fusion can reduce data movement overhead and improve inference performance. This PR adds fused `conv2d_add` op for onednn backend, which will be used for int8 inference with onednn backend. Cannot call this op with other quantization backends otherwise an error is thrown. **Test Plan** ``` python -m pytest test_quantization.py::TestQuantizedConv ``` cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

leslie-fang-intel · 2023-01-28T03:21:11Z

@pytorchbot merge

pytorchmergebot · 2023-01-28T03:23:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[Quant] Add fused conv_add op for onednn backend

6e3cdbe

[ghstack-poisoned]

leslie-fang-intel requested review from jerryzh168, z-a-f, salilsdesai, kimishpatel, digantdesai and jianyuh as code owners December 6, 2022 06:25

pytorch-bot bot added the release notes: quantization release notes category label Dec 6, 2022

leslie-fang-intel mentioned this pull request Dec 6, 2022

[Quant] Remove all the dequant nodes when the ref module has multi input args #90157

Closed

leslie-fang-intel added a commit that referenced this pull request Dec 6, 2022

[Quant] Add fused conv_add op for onednn backend

947fabf

ghstack-source-id: ca56717ef6b01387b769bd7b679b3e82d5d696b4 Pull Request resolved: #90262

github-actions bot added module: cpu CPU specific problem (e.g., perf, algorithm) module: mkldnn Related to Intel IDEEP or oneDNN (a.k.a. mkldnn) integration labels Dec 6, 2022

leslie-fang-intel marked this pull request as draft December 6, 2022 06:26

pytorchbot added the open source label Dec 6, 2022

leslie-fang-intel changed the title ~~[Quant] Add fused conv_add op for onednn backend~~ [WIP] [Quant] Add fused conv_add op for onednn backend Dec 6, 2022

Update on "[WIP] [Quant] Add fused conv_add op for onednn backend"

8c2cd94

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

leslie-fang-intel added a commit that referenced this pull request Dec 6, 2022

[Quant] Add fused conv_add op for onednn backend

aecd0dd

ghstack-source-id: a7f5fb324870a2dbdecb84ed9ee76446f6450002 Pull Request resolved: #90262

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 7, 2022

[Quant] Add fused conv_add op for onednn backend

c8f5a32

ghstack-source-id: a7f5fb324870a2dbdecb84ed9ee76446f6450002 Pull Request resolved: pytorch#90262

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 7, 2022

[Quant] Add fused conv_add op for onednn backend

ce88345

ghstack-source-id: a7f5fb324870a2dbdecb84ed9ee76446f6450002 Pull Request resolved: pytorch#90262

Update on "[WIP] [Quant] Add fused conv_add op for onednn backend"

3b1bfae

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 gujinghui PenghuiCheng jianyuh min-jean-cho yanbing-j Guobing-Chen Xia-Weiwen [ghstack-poisoned]

leslie-fang-intel added a commit that referenced this pull request Dec 7, 2022

[Quant] Add fused conv_add op for onednn backend

43e9c8a

ghstack-source-id: 324e97d3dcda37d6abad83af61453aaf0e9d209e Pull Request resolved: #90262

leslie-fang-intel changed the title ~~[WIP] [Quant] Add fused conv_add op for onednn backend~~ [Quant] Add fused conv_add op for onednn backend Dec 7, 2022

leslie-fang-intel added intel This tag is for PR from Intel ciflow/trunk Trigger trunk jobs on your pull request labels Dec 7, 2022

leslie-fang-intel requested review from Xia-Weiwen, jgong5 and XiaobingSuper December 7, 2022 02:14

leslie-fang-intel changed the title ~~[Quant] Add fused conv_add op for onednn backend~~ [Quant] Add fused conv2d_add op for onednn backend Dec 7, 2022

leslie-fang-intel mentioned this pull request Dec 7, 2022

[Quant] Add fused conv2d_add_relu op for onednn backend #90364

Closed

This was referenced Dec 20, 2022

[Quant][FX] Lower QConvAdd2d for onednn backend #91153

Closed

[Quant] Add fused ConvAddReLU2d module for onednn backend #91154

Closed

[Quant][FX] Lower QConvAddReLU2d for onednn backend #91155

Closed

leslie-fang-intel added 2 commits December 20, 2022 15:11

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Dec 20, 2022

[Quant] Add fused conv_add op for onednn backend

3a420e1

ghstack-source-id: 2b22fc9ffcf1350eac41ce219dfe6d3110abd668 Pull Request resolved: pytorch#90262

leslie-fang-intel added 4 commits January 4, 2023 10:22

jerryzh168 reviewed Jan 9, 2023

View reviewed changes

jerryzh168 approved these changes Jan 9, 2023

View reviewed changes

jerryzh168 reviewed Jan 9, 2023

View reviewed changes

leslie-fang-intel assigned leslie-fang-intel and unassigned leslie-fang-intel Jan 10, 2023

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Jan 13, 2023

[Quant] Add fused conv_add op for onednn backend

15320c0

ghstack-source-id: bb2010eccb3737ed8d8706fa43c7c24982a64b72 Pull Request resolved: pytorch#90262

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Jan 26, 2023

[Quant] Add fused conv_add op for onednn backend

5b211b5

ghstack-source-id: bb2010eccb3737ed8d8706fa43c7c24982a64b72 Pull Request resolved: pytorch#90262

leslie-fang-intel added 5 commits January 26, 2023 10:14

pytorchmergebot added the Merged label Jan 28, 2023

pytorchmergebot closed this in a62fc09 Jan 28, 2023

facebook-github-bot deleted the gh/leslie-fang-intel/4/head branch June 8, 2023 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] Add fused conv2d_add op for onednn backend #90262

[Quant] Add fused conv2d_add op for onednn backend #90262

leslie-fang-intel commented Dec 6, 2022 •

edited

pytorch-bot bot commented Dec 6, 2022 •

edited

jerryzh168 Jan 9, 2023

leslie-fang-intel Jan 10, 2023

jerryzh168 Jan 9, 2023

leslie-fang-intel Jan 10, 2023

jerryzh168 Jan 9, 2023 •

edited

leslie-fang-intel Jan 10, 2023

leslie-fang-intel commented Jan 28, 2023

pytorchmergebot commented Jan 28, 2023

[Quant] Add fused conv2d_add op for onednn backend #90262

[Quant] Add fused conv2d_add op for onednn backend #90262

Conversation

leslie-fang-intel commented Dec 6, 2022 • edited

pytorch-bot bot commented Dec 6, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90262

✅ No Failures

jerryzh168 Jan 9, 2023

Choose a reason for hiding this comment

leslie-fang-intel Jan 10, 2023

Choose a reason for hiding this comment

jerryzh168 Jan 9, 2023

Choose a reason for hiding this comment

leslie-fang-intel Jan 10, 2023

Choose a reason for hiding this comment

jerryzh168 Jan 9, 2023 • edited

Choose a reason for hiding this comment

leslie-fang-intel Jan 10, 2023

Choose a reason for hiding this comment

leslie-fang-intel commented Jan 28, 2023

pytorchmergebot commented Jan 28, 2023

Merge started

leslie-fang-intel commented Dec 6, 2022 •

edited

pytorch-bot bot commented Dec 6, 2022 •

edited

jerryzh168 Jan 9, 2023 •

edited