[Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer #111280

leslie-fang-intel · 2023-10-14T07:21:13Z

Stack from ghstack (oldest at bottom):

Summary
This PR enables PT2 QAT Quantization flow in X86InductorQuantizer.

Test Plan

python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api
python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api
python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d
python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

[ghstack-poisoned]

pytorch-bot · 2023-10-14T07:21:17Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/111280

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e73c3ab with merge base a126bbf ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: cfa1febfc4ec56317a847c31bc984e86eb696e2b Pull Request resolved: pytorch#111280

…uantizer" **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

leslie-fang-intel · 2023-10-23T10:40:06Z

Hi @jerryzh168 @andrewor14, Could you kindly help to take a look of this PR?
One thing to note is per our discussion previously, we prefer small test case since it's easy to debug. So I write separate test case for QAT and PTQ, even some of UTs has same pattern to check as PTQ.

test/inductor/test_mkldnn_pattern_matcher.py

jerryzh168 · 2023-10-31T16:48:13Z

test/inductor/test_mkldnn_pattern_matcher.py

+            4,
+            17,


these numbers will be tricky to get right I feel, so ideally this can be structured differently, e.g. like specifying a list of nodes etc.

Yean, it's tricky to make it correct. I add a following up PR to re-structure it: #112570 in this ghstack. After that, we only need to check the patterns we care in each test case such as: dequant promotion, QConv/Linear Unary and QConv Binary instead of all the patterns . Please help to take a look of that PR also @jerryzh168.

jerryzh168 · 2023-10-31T16:50:50Z

torch/ao/quantization/quantizer/x86_inductor_quantizer.py

+            if (
+                conv_node.op != "call_function"
+                or conv_node.target != torch.ops.aten.conv2d.default
+            ):


just making sure, do you have plans to follow https://pytorch.org/tutorials/prototype/pt2e_quantizer.html#a-note-on-ir-for-pt2e-quantization-flow later?

Did you mean use SubgraphMatcherWithNameNodeMap to match the pattern for quantizer? I think we can follow up the changes later, maybe after the changes in xnnpack_quantizer. BTW: as discussed in the slack, is there any plan to support functionalization for pre autograd aten IR? I think it can help us save lots of patterns to use SubgraphMatcherWithNameNodeMap .

sg, yeah functionalization will be supported in 1-2 month I think

jerryzh168 · 2023-10-31T16:52:38Z

torch/ao/quantization/quantizer/x86_inductor_quantizer.py

+        self, model: torch.fx.GraphModule, config: QuantizationConfig
+    ):
+        # Annotate QAT Specific patterns
+        self._annotate_qat_conv2d_bn_unary(model, config)


do you need to add something to prepare_pt2e code to support this pattern? https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/quantize_pt2e.py#L178

we are also thinking about moving qat specific code to quantizer as well actually, we'll discuss a bit internally and get your feedback

Yean, for now we only have relu as unary post op which I think is already supported in https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/quantize_pt2e.py#L178.

ghstack-source-id: 1e15fb51e52a95ef9e9d54bc01477a031c1dec70 Pull Request resolved: pytorch#111280

…uantizer" **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

leslie-fang-intel · 2023-11-01T08:04:48Z

Hi @jerryzh168 @andrewor14, thanks for the comments. Please kindly help to take a look again.

…uantizer" **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: 70c4f4f2f04154fe0dcef61e844f70874f14912c Pull Request resolved: pytorch#111280

leslie-fang-intel · 2023-11-02T02:01:08Z

@pytorchbot merge

pytorchmergebot · 2023-11-02T02:02:52Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

#111281) **Summary** This PR adds ConvBNAdd(ReLU) QAT Annotation into `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add_relu ``` Pull Request resolved: #111281 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: #111280

…ytorch#111280) **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` Pull Request resolved: pytorch#111280 Approved by: https://github.com/jgong5, https://github.com/jerryzh168

pytorch#111281) **Summary** This PR adds ConvBNAdd(ReLU) QAT Annotation into `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add_relu ``` Pull Request resolved: pytorch#111281 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: pytorch#111280

…ytorch#111280) **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` Pull Request resolved: pytorch#111280 Approved by: https://github.com/jgong5, https://github.com/jerryzh168

pytorch#111281) **Summary** This PR adds ConvBNAdd(ReLU) QAT Annotation into `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add_relu ``` Pull Request resolved: pytorch#111281 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: pytorch#111280

…ytorch#111280) **Summary** This PR enables PT2 QAT Quantization flow in `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_relu ``` Pull Request resolved: pytorch#111280 Approved by: https://github.com/jgong5, https://github.com/jerryzh168

pytorch#111281) **Summary** This PR adds ConvBNAdd(ReLU) QAT Annotation into `X86InductorQuantizer`. **Test Plan** ``` python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_with_quantizer_api python -m pytest test_x86inductor_quantizer.py -k test_qat_conv2d_binary_unary_with_quantizer_api python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_qconv2d_add_relu ``` Pull Request resolved: pytorch#111281 Approved by: https://github.com/jgong5, https://github.com/jerryzh168 ghstack dependencies: pytorch#111280

add ConvBN(ReLU) recipe

26c5dc1

[ghstack-poisoned]

leslie-fang-intel requested a review from jerryzh168 as a code owner October 14, 2023 07:21

pytorch-bot bot added the release notes: quantization release notes category label Oct 14, 2023

leslie-fang-intel mentioned this pull request Oct 14, 2023

[Quant] [PT2] Add ConvBNAdd(ReLU) Annotation into X86InductorQuantizer #111281

Closed

github-actions bot added the module: inductor label Oct 14, 2023

leslie-fang-intel changed the title ~~add ConvBN(ReLU) recipe~~ [Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer Oct 14, 2023

leslie-fang-intel added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 14, 2023

leslie-fang-intel requested review from jgong5 and andrewor14 October 14, 2023 07:27

pytorchbot added the open source label Oct 14, 2023

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Oct 17, 2023

add ConvBN(ReLU) recipe

e68515d

ghstack-source-id: cfa1febfc4ec56317a847c31bc984e86eb696e2b Pull Request resolved: pytorch#111280

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Oct 21, 2023

add ConvBN(ReLU) recipe

c49af8a

ghstack-source-id: cfa1febfc4ec56317a847c31bc984e86eb696e2b Pull Request resolved: pytorch#111280

jgong5 approved these changes Oct 23, 2023

View reviewed changes

jerryzh168 reviewed Oct 31, 2023

View reviewed changes

test/inductor/test_mkldnn_pattern_matcher.py Outdated Show resolved Hide resolved

jerryzh168 reviewed Oct 31, 2023

View reviewed changes

test/inductor/test_mkldnn_pattern_matcher.py Outdated Show resolved Hide resolved

jerryzh168 reviewed Oct 31, 2023

View reviewed changes

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 1, 2023

add ConvBN(ReLU) recipe

a1d60ef

ghstack-source-id: 1e15fb51e52a95ef9e9d54bc01477a031c1dec70 Pull Request resolved: pytorch#111280

leslie-fang-intel mentioned this pull request Nov 1, 2023

[Inductor] [Quant] Re-structure Quantization testcase pattern matcher check #112570

Closed

leslie-fang-intel requested a review from jerryzh168 November 1, 2023 08:03

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 1, 2023

add ConvBN(ReLU) recipe

d5b20d7

ghstack-source-id: 70c4f4f2f04154fe0dcef61e844f70874f14912c Pull Request resolved: pytorch#111280

jerryzh168 approved these changes Nov 1, 2023

View reviewed changes

pytorchmergebot added the merging label Nov 2, 2023

pytorchmergebot added Merged and removed merging labels Nov 2, 2023

pytorchmergebot closed this in 56ca004 Nov 2, 2023

facebook-github-bot deleted the gh/leslie-fang-intel/31/head branch November 5, 2023 15:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer #111280

[Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer #111280

leslie-fang-intel commented Oct 14, 2023 •

edited

pytorch-bot bot commented Oct 14, 2023 •

edited

leslie-fang-intel commented Oct 23, 2023

jerryzh168 Oct 31, 2023

leslie-fang-intel Nov 1, 2023

jerryzh168 Oct 31, 2023

leslie-fang-intel Nov 1, 2023

jerryzh168 Nov 1, 2023

jerryzh168 Oct 31, 2023

leslie-fang-intel Nov 1, 2023

leslie-fang-intel commented Nov 1, 2023

leslie-fang-intel commented Nov 2, 2023

pytorchmergebot commented Nov 2, 2023

[Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer #111280

[Quant] [PT2] Enable QAT Quantization flow in X86InductorQuantizer #111280

Conversation

leslie-fang-intel commented Oct 14, 2023 • edited

pytorch-bot bot commented Oct 14, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/111280

✅ No Failures

leslie-fang-intel commented Oct 23, 2023

jerryzh168 Oct 31, 2023

Choose a reason for hiding this comment

leslie-fang-intel Nov 1, 2023

Choose a reason for hiding this comment

jerryzh168 Oct 31, 2023

Choose a reason for hiding this comment

leslie-fang-intel Nov 1, 2023

Choose a reason for hiding this comment

jerryzh168 Nov 1, 2023

Choose a reason for hiding this comment

jerryzh168 Oct 31, 2023

Choose a reason for hiding this comment

leslie-fang-intel Nov 1, 2023

Choose a reason for hiding this comment

leslie-fang-intel commented Nov 1, 2023

leslie-fang-intel commented Nov 2, 2023

pytorchmergebot commented Nov 2, 2023

Merge started

leslie-fang-intel commented Oct 14, 2023 •

edited

pytorch-bot bot commented Oct 14, 2023 •

edited