Enable oneDNN QConv FP32/BF16 output #112010

leslie-fang-intel · 2023-10-25T08:34:01Z

Stack from ghstack (oldest at bottom):

Summary

PR 1 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor [RFC] Enable Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor #111640.
Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output.

Test Plan

python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e
python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e
python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e
python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e
python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e
python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e
python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

[ghstack-poisoned]

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

pytorch-bot · 2023-10-25T08:34:07Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/112010

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit c13ee83 with merge base 0d95378 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

trunk / macos-12-py3-arm64-mps / test (mps, 1, 1, macos-m1-12) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: c2cb5dafb6dc1cb9e3df891305b0e4d72a7f8fd1 Pull Request resolved: #112010

ghstack-source-id: e970db7531249b5b9ed34fba2920dbe3cdc85e21 Pull Request resolved: #112010

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: 7761679717957944c7955fc5cc980788c411102f Pull Request resolved: #112010

**Summary** Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: fc5934481d0eef3772f9fe1123986b484c592793 Pull Request resolved: #112010

**Summary** Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: 922de9b84b66fa1166d42344f3a8e70d47c79668 Pull Request resolved: #112010

aten/src/ATen/native/quantized/cpu/qconv.cpp

**Summary** - PR 1 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor #111640. - Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: af28fba9e9a2d6a03f9efe18496a9c173143ecd2 Pull Request resolved: pytorch#112010

**Summary** - PR 1 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor #111640. - Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: 452927ed3a4125a3e817749bed4ae57cab0afe24 Pull Request resolved: pytorch#112010

leslie-fang-intel · 2023-11-03T08:14:18Z

@pytorchbot merge

pytorchmergebot · 2023-11-03T08:16:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

**Summary** - PR 2 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor #111640. - Enable QLinear (relu) with BFloat16 or Float32 output. **TestPlan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qlinear_pt2e ``` Pull Request resolved: #112126 Approved by: https://github.com/jerryzh168, https://github.com/jgong5 ghstack dependencies: #112010

…torQuantizer (#112140) **Summary** - PR 3 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor #111640. - Remove the output annotation of QConv/QLinear in X86InductorQuantizer. **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qlinear python -m pytest test_x86inductor_quantizer.py -k Conv2d python -m pytest test_x86inductor_quantizer.py -k Linear ``` Pull Request resolved: #112140 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: #112010, #112126

**Summary** - PR 1 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` Pull Request resolved: pytorch#112010 Approved by: https://github.com/jerryzh168, https://github.com/jgong5

**Summary** - PR 2 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Enable QLinear (relu) with BFloat16 or Float32 output. **TestPlan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qlinear_pt2e ``` Pull Request resolved: pytorch#112126 Approved by: https://github.com/jerryzh168, https://github.com/jgong5 ghstack dependencies: pytorch#112010

…torQuantizer (pytorch#112140) **Summary** - PR 3 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Remove the output annotation of QConv/QLinear in X86InductorQuantizer. **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qlinear python -m pytest test_x86inductor_quantizer.py -k Conv2d python -m pytest test_x86inductor_quantizer.py -k Linear ``` Pull Request resolved: pytorch#112140 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: pytorch#112010, pytorch#112126

**Summary** - PR 1 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Enable QConv (relu, add, add_relu) with BFloat16 or Float32 output. **Test Plan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qconv1d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv2d_pt2e python -u -m pytest -s -v test_quantized_op.py -k test_qconv3d_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_pt2e python -u -m pytest test_quantized_op.py -k test_qconv2d_add_relu_float_output_pt2e ``` Pull Request resolved: pytorch#112010 Approved by: https://github.com/jerryzh168, https://github.com/jgong5

**Summary** - PR 2 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Enable QLinear (relu) with BFloat16 or Float32 output. **TestPlan** ``` python -u -m pytest -s -v test_quantized_op.py -k test_qlinear_pt2e ``` Pull Request resolved: pytorch#112126 Approved by: https://github.com/jerryzh168, https://github.com/jgong5 ghstack dependencies: pytorch#112010

…torQuantizer (pytorch#112140) **Summary** - PR 3 for enabling Int8-Mixed-BF16 PT2E PTQ Quantization with Inductor pytorch#111640. - Remove the output annotation of QConv/QLinear in X86InductorQuantizer. **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qlinear python -m pytest test_x86inductor_quantizer.py -k Conv2d python -m pytest test_x86inductor_quantizer.py -k Linear ``` Pull Request resolved: pytorch#112140 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: pytorch#112010, pytorch#112126

leslie-fang-intel added 2 commits October 25, 2023 16:30

Enable onednn.QConv FP32/BF16 output

71fa799

[ghstack-poisoned]

leslie-fang-intel requested review from jerryzh168, salilsdesai, kimishpatel, digantdesai and jianyuh as code owners October 25, 2023 08:34

pytorch-bot bot added the release notes: quantization release notes category label Oct 25, 2023

github-actions bot added module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor ciflow/inductor labels Oct 25, 2023

leslie-fang-intel added a commit that referenced this pull request Oct 25, 2023

Enable onednn.QConv FP32/BF16 output

2d2c6f4

ghstack-source-id: c2cb5dafb6dc1cb9e3df891305b0e4d72a7f8fd1 Pull Request resolved: #112010

leslie-fang-intel marked this pull request as draft October 25, 2023 08:34

leslie-fang-intel changed the title ~~Enable onednn.QConv FP32/BF16 output~~ [WIP] Enable onednn.QConv FP32/BF16 output Oct 25, 2023

leslie-fang-intel added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 25, 2023

leslie-fang-intel added a commit that referenced this pull request Oct 25, 2023

Enable onednn.QConv FP32/BF16 output

6aa214a

ghstack-source-id: e970db7531249b5b9ed34fba2920dbe3cdc85e21 Pull Request resolved: #112010

leslie-fang-intel changed the title ~~[WIP] Enable onednn.QConv FP32/BF16 output~~ [WIP] Enable onednn QConv FP32/BF16 output Oct 25, 2023

pytorchbot added the open source label Oct 25, 2023

leslie-fang-intel changed the title ~~[WIP] Enable onednn QConv FP32/BF16 output~~ Enable onednn QConv FP32/BF16 output Oct 26, 2023

leslie-fang-intel added a commit that referenced this pull request Oct 26, 2023

Enable onednn.QConv FP32/BF16 output

ba27d4f

ghstack-source-id: 7761679717957944c7955fc5cc980788c411102f Pull Request resolved: #112010

leslie-fang-intel added a commit that referenced this pull request Oct 26, 2023

Enable onednn.QConv FP32/BF16 output

f36336f

ghstack-source-id: fc5934481d0eef3772f9fe1123986b484c592793 Pull Request resolved: #112010

leslie-fang-intel added a commit that referenced this pull request Oct 26, 2023

Enable onednn.QConv FP32/BF16 output

e74aa71

ghstack-source-id: 922de9b84b66fa1166d42344f3a8e70d47c79668 Pull Request resolved: #112010

This was referenced Oct 26, 2023

Enable oneDNN QLinear FP32/BF16 output #112126

Closed

[Quant] [PT2] Remove the output Annotation of Conv/Linear in x86InductorQuantizer #112140

Closed

[Quant] [PT2] Enable Decomposed quant per tensor/channel to accept bfloat16 input #112225

Closed

jerryzh168 approved these changes Oct 31, 2023

View reviewed changes

leslie-fang-intel changed the title ~~Enable onednn QConv FP32/BF16 output~~ Enable oneDNN QConv FP32/BF16 output Oct 31, 2023

This was referenced Oct 31, 2023

[Inductor] [Quant] Enable QConv2d int8-mixed-bf16 Lowering #112469

Closed

[Inductor] [Quant] Enable QLinear int8-mixed-bf16 Lowering #112486

Closed

leslie-fang-intel requested review from jgong5 and Xia-Weiwen November 1, 2023 02:13

This was referenced Nov 1, 2023

[Inductor] [Quant] Enable QConv2d Unary int8-mixed-bf16 Lowering #112550

Closed

[Inductor] [Quant] Enable QConv2d Binary int8-mixed-bf16 Lowering #112551

Closed

leslie-fang-intel marked this pull request as ready for review November 1, 2023 02:14

jgong5 approved these changes Nov 1, 2023

View reviewed changes

aten/src/ATen/native/quantized/cpu/qconv.cpp Outdated Show resolved Hide resolved

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 3, 2023

Enable onednn.QConv FP32/BF16 output

498d6d7

ghstack-source-id: af28fba9e9a2d6a03f9efe18496a9c173143ecd2 Pull Request resolved: pytorch#112010

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 3, 2023

Enable onednn.QConv FP32/BF16 output

efacc46

ghstack-source-id: af28fba9e9a2d6a03f9efe18496a9c173143ecd2 Pull Request resolved: pytorch#112010

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 3, 2023

Enable onednn.QConv FP32/BF16 output

6a3b887

ghstack-source-id: 452927ed3a4125a3e817749bed4ae57cab0afe24 Pull Request resolved: pytorch#112010

pytorchmergebot added the merging label Nov 3, 2023

pytorchmergebot added the Merged label Nov 3, 2023

pytorchmergebot closed this in b6fc7af Nov 3, 2023

pytorchmergebot removed the merging label Nov 3, 2023

facebook-github-bot deleted the gh/leslie-fang-intel/33/head branch November 6, 2023 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable oneDNN QConv FP32/BF16 output #112010

Enable oneDNN QConv FP32/BF16 output #112010

leslie-fang-intel commented Oct 25, 2023 •

edited

pytorch-bot bot commented Oct 25, 2023 •

edited

leslie-fang-intel commented Nov 3, 2023

pytorchmergebot commented Nov 3, 2023

Enable oneDNN QConv FP32/BF16 output #112010

Enable oneDNN QConv FP32/BF16 output #112010

Conversation

leslie-fang-intel commented Oct 25, 2023 • edited

pytorch-bot bot commented Oct 25, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/112010

✅ You can merge normally! (1 Unrelated Failure)

leslie-fang-intel commented Nov 3, 2023

pytorchmergebot commented Nov 3, 2023

Merge started

leslie-fang-intel commented Oct 25, 2023 •

edited

pytorch-bot bot commented Oct 25, 2023 •

edited