[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854

Le-Zheng · 2023-11-30T07:53:10Z

Stack from ghstack (oldest at bottom):

[Quant][Inductor] Enable quantization linear pattern fusion with int8_mixed_bf16 for gelu #116004
-> [Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854
[Quant][PT2E] Enable linear and linear-unary post-op gelu quant recipe for x86 inductor quantizer #114853

Summary
Enable QLinear Unary pattern for gelu with int8

Test plan
python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @amjames @desertfire @chauhang

…inside inductor [ghstack-poisoned]

pytorch-bot · 2023-11-30T07:53:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114854

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f19e007 with merge base cb489e7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…inside inductor ghstack-source-id: 10f4c46 Pull Request resolved: #114854

torch/_inductor/fx_passes/quantization.py

leslie-fang-intel · 2023-11-30T12:29:04Z

torch/_inductor/fx_passes/quantization.py

+                _gelu_fusion_1(get_qlinear(2)),
+                dtype=original_pattern_output_dtype,
+            ),
+            UnaryAttr("gelu", [], ""): generate_pattern_with_output_quant(


This UnaryAttr("gelu", [], "") should have different algorithm_attr as above, right?

I believe we should set the algorithm_attr for GeLU here to distinguish 2 different algorithms.

Thank you for your feedback! I've made the requested changes.

leslie-fang-intel · 2023-11-30T12:37:12Z

test/inductor/test_mkldnn_pattern_matcher.py

                super().__init__()
                self.linear = torch.nn.Linear(4, 4, use_bias)
-                self.unary_fn = torch.nn.ReLU()
+                if postop == torch.nn.GELU:
+                    self.unary_fn = postop(approximate=post_op_algo)


Looks like you already pass in a instance of torch.nn.GELU(), I am afraid it can't be initialized here again?

You are right! Thanks for the comment and changed accordingly.

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_unary cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

…inside inductor ghstack-source-id: f4986f9 Pull Request resolved: #114854

jgong5 · 2023-12-01T04:53:42Z

torch/_inductor/fx_passes/quantization.py

@@ -496,6 +517,8 @@ def qconv_binary(match: Match, *args, **kwargs):


 def _register_quantization_unary_fusion():
+    from .mkldnn_fusion import _gelu_fusion_1, _gelu_fusion_2


_gelu_fusion_1 is for erf and _gelu_fusion_2 is for tanh approximation? If so, can we give more meaningful name to them instead of "1" and "2"?

Thank you for your feedback! I use _gelu_fusion_erf and _gelu_fusion_tanh instead.

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_unary cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

…usion for gelu inside inductor ghstack-source-id: f42e8f2 Pull Request resolved: #114854

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_unary cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

…antization linear pattern fusion for gelu inside inductor ghstack-source-id: 7fa14d7 Pull Request resolved: #114854

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_unary cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

leslie-fang-intel · 2023-12-19T07:22:21Z

test/inductor/test_mkldnn_pattern_matcher.py

                self.linear2 = torch.nn.Linear(4, 4, use_bias)
-                self.unary_fn2 = torch.nn.ReLU()
+                if unary_op == torch.nn.GELU():


2 different GELU instances should not be same always.

Thanks for the comment and changed accordingly.

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

… for gelu inside inductor ghstack-source-id: 458d785 Pull Request resolved: pytorch#114854

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

… fusion for gelu inside inductor ghstack-source-id: c7ffc92 Pull Request resolved: pytorch#114854

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

… for gelu inside inductor ghstack-source-id: 3e91151 Pull Request resolved: pytorch#114854

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

torch/_inductor/fx_passes/quantization.py

…n for gelu inside inductor" **Summary** Enable QLinear Unary pattern for gelu with int8 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_cpu cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

github-actions · 2024-03-02T05:34:02Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[ghstack-poisoned]

leslie-fang-intel · 2024-03-14T01:47:12Z

@pytorchbot merge

pytorchmergebot · 2024-03-14T01:49:00Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…_mixed_bf16 for gelu (#116004) **Summary** Enable QLinear Unary pattern for gelu with int8_mix_bf16 **Test plan** python test/inductor/test_mkldnn_pattern_matcher.py -k test_qlinear_gelu_int8_mixed_bf16 Co-authored-by: leslie-fang-intel <leslie.fang@intel.com> Pull Request resolved: #116004 Approved by: https://github.com/jgong5, https://github.com/leslie-fang-intel ghstack dependencies: #114853, #114854

[Quant][Inductor] Enable quantization linear pattern fusion for gelu …

baa4aa9

…inside inductor [ghstack-poisoned]

This was referenced Nov 30, 2023

[Quant] Add int8 linear op gelu for quantization PT2E with Inductor. input is an int8 CPU tensor; weight is an int8 MdkldnnCPU tensor #114852

Closed

[Quant][PT2E] Enable linear and linear-unary post-op gelu quant recipe for x86 inductor quantizer #114853

Closed

github-actions bot added module: inductor ciflow/inductor labels Nov 30, 2023

Le-Zheng added a commit that referenced this pull request Nov 30, 2023

[Quant][Inductor] Enable quantization linear pattern fusion for gelu …

fabcbd5

…inside inductor ghstack-source-id: 10f4c46 Pull Request resolved: #114854

pytorchbot added the open source label Nov 30, 2023

Le-Zheng mentioned this pull request Nov 30, 2023

[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114101

Closed

Le-Zheng requested review from leslie-fang-intel and jgong5 and removed request for leslie-fang-intel November 30, 2023 09:35

leslie-fang-intel reviewed Nov 30, 2023

View reviewed changes

torch/_inductor/fx_passes/quantization.py Outdated Show resolved Hide resolved

leslie-fang-intel reviewed Nov 30, 2023

View reviewed changes

leslie-fang-intel requested changes Nov 30, 2023

View reviewed changes

Le-Zheng added a commit that referenced this pull request Dec 1, 2023

[Quant][Inductor] Enable quantization linear pattern fusion for gelu …

ea70df3

…inside inductor ghstack-source-id: f4986f9 Pull Request resolved: #114854

jgong5 reviewed Dec 1, 2023

View reviewed changes

Le-Zheng added a commit that referenced this pull request Dec 1, 2023

address comment[Quant][Inductor] Enable quantization linear pattern f…

92c2cce

…usion for gelu inside inductor ghstack-source-id: f42e8f2 Pull Request resolved: #114854

Le-Zheng added a commit that referenced this pull request Dec 4, 2023

instance get_qlinear in ut address comment[Quant][Inductor] Enable qu…

022fa58

…antization linear pattern fusion for gelu inside inductor ghstack-source-id: 7fa14d7 Pull Request resolved: #114854

leslie-fang-intel reviewed Dec 19, 2023

View reviewed changes

Le-Zheng added 2 commits December 18, 2023 23:54

Le-Zheng added a commit to Le-Zheng/pytorch that referenced this pull request Dec 20, 2023

update UT [Quant][Inductor] Enable quantization linear pattern fusion…

1811044

… for gelu inside inductor ghstack-source-id: 458d785 Pull Request resolved: pytorch#114854

Le-Zheng added a commit to Le-Zheng/pytorch that referenced this pull request Dec 20, 2023

rebase update UT [Quant][Inductor] Enable quantization linear pattern…

d720e6a

… fusion for gelu inside inductor ghstack-source-id: c7ffc92 Pull Request resolved: pytorch#114854

Le-Zheng added a commit to Le-Zheng/pytorch that referenced this pull request Dec 21, 2023

update UT [Quant][Inductor] Enable quantization linear pattern fusion…

e8de6c1

… for gelu inside inductor ghstack-source-id: 3e91151 Pull Request resolved: pytorch#114854

Le-Zheng requested a review from leslie-fang-intel December 22, 2023 02:24

leslie-fang-intel approved these changes Dec 22, 2023

View reviewed changes

Le-Zheng requested a review from jgong5 December 22, 2023 07:55

jgong5 approved these changes Dec 22, 2023

View reviewed changes

torch/_inductor/fx_passes/quantization.py Outdated Show resolved Hide resolved

github-actions bot added the Stale label Mar 2, 2024

Update

c4f1da1

[ghstack-poisoned]

leslie-fang-intel added ciflow/trunk Trigger trunk jobs on your pull request topic: not user facing topic category and removed Stale labels Mar 13, 2024

Update

f19e007

[ghstack-poisoned]

pytorchmergebot added the merging label Mar 14, 2024

pytorchmergebot closed this in 43d68e9 Mar 14, 2024

pytorchmergebot added Merged and removed merging labels Mar 14, 2024

github-actions bot deleted the gh/le-zheng/10/head branch April 14, 2024 02:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854

[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854

Uh oh!

Le-Zheng commented Nov 30, 2023 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Nov 30, 2023 •

edited

Loading

Uh oh!

Uh oh!

leslie-fang-intel Nov 30, 2023

Uh oh!

leslie-fang-intel Nov 30, 2023

Uh oh!

Le-Zheng Dec 6, 2023

Uh oh!

leslie-fang-intel Nov 30, 2023

Uh oh!

Le-Zheng Dec 22, 2023

Uh oh!

jgong5 Dec 1, 2023

Uh oh!

Le-Zheng Dec 6, 2023

Uh oh!

leslie-fang-intel Dec 19, 2023

Uh oh!

Le-Zheng Dec 22, 2023

Uh oh!

Uh oh!

github-actions bot commented Mar 2, 2024

Uh oh!

leslie-fang-intel commented Mar 14, 2024

Uh oh!

pytorchmergebot commented Mar 14, 2024

Uh oh!

Uh oh!

		@@ -496,6 +517,8 @@ def qconv_binary(match: Match, args, *kwargs):


		def _register_quantization_unary_fusion():
		from .mkldnn_fusion import _gelu_fusion_1, _gelu_fusion_2

[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854

[Quant][Inductor] Enable quantization linear pattern fusion for gelu inside inductor #114854

Uh oh!

Conversation

Le-Zheng commented Nov 30, 2023 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114854

✅ No Failures

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 2, 2024

Uh oh!

leslie-fang-intel commented Mar 14, 2024

Uh oh!

pytorchmergebot commented Mar 14, 2024

Merge started

Uh oh!

Uh oh!

Le-Zheng commented Nov 30, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 30, 2023 •

edited

Loading