inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) #97007

XiaobingSuper · 2023-03-17T07:29:44Z

Stack from ghstack (oldest at bottom):

cc @jgong5 @mingfeima @sanchitintel @ashokei @jingxu10 @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire

[ghstack-poisoned]

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

pytorch-bot · 2023-03-17T07:29:47Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/97007

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 186a588:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 779777fad1d385b53a45524b12c3401f787277d5 Pull Request resolved: #97007

ghstack-source-id: ae74b1f32376fe8941ea8c8e3e01e3f0f45bd470 Pull Request resolved: #97007

ghstack-source-id: e5f15609e22d0171b3945f0f27aff7039170815e Pull Request resolved: #97007

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

ghstack-source-id: b6ab144c346a478f87e8f7b77cf35ea55ea05500 Pull Request resolved: #97007

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

ghstack-source-id: 9484d4056c62d3cd1cc3c0e501c4410977f589ee Pull Request resolved: #97007

jgong5

LGTM.

torch/_inductor/pattern_matcher.py

EikanWang · 2023-03-20T05:25:18Z

torch/_inductor/mkldnn.py

@@ -138,7 +130,7 @@ def _update_module_params(self, conv, unary, input_size):

    def _conv_forward(self, input, weight, bias):
        if self.padding_mode != "zeros":
-            return torch.ops.mkldnn._convolution_pointwise(
+            return torch.mkldnn_convolution(


Let's add a single conv and linear for the repacked conv. The semantic of _conv_pointwise is confusing as it serves both the single conv and the fused conv.

torch/_inductor/pattern_matcher.py

EikanWang · 2023-03-20T05:31:47Z

torch/_inductor/pattern_matcher.py

+                return L[aten.clamp_max](
+                    L[aten.clamp_min](conv_out, min_value), max_value
+                )


This is the decompose version of hardtanh, right? Is there any way to call the decomposed function directly but not duplicate it again.

I didn't find a way to do it.

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…onv_unary)" cc jgong5 mingfeima sanchitintel ashokei jingxu10 soumith voznesenskym penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

pytorchmergebot · 2023-03-30T16:46:53Z

Rebased gh/XiaobingSuper/84/orig onto refs/remotes/origin/viable/strict because #97141 was rebased, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/97007)

…onv_unary)" cc jgong5 mingfeima sanchitintel ashokei jingxu10 soumith voznesenskym penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

desertfire · 2023-04-07T01:58:20Z

torch/_inductor/pattern_matcher.py

@@ -608,6 +608,279 @@ def addmm(match, mat1, mat2, inp):
        return L[aten.add](inp, L[aten.mm](mat1, mat2))


+# TODO(XiaobingSuper): move it to fx_passes/mkldnn_fuse.py


I am ok with this stack of changes, but please move them into a separate file.

You can create a fx_passes/ folder to match with #97741

Yes, move it to a separate file.

jansel · 2023-04-07T14:52:09Z

torch/_inductor/pattern_matcher.py

+    def relu_fusion(computation_call):
+        return CallFunction(aten.relu, computation_call)
+
+    def sigmoid_fusion(computation_call):
+        return CallFunction(aten.sigmoid, computation_call)
+
+    def tanh_fusion(computation_call):
+        return CallFunction(aten.tanh, computation_call)


You can match multiple functions in a single pattern like:

Suggested change

def relu_fusion(computation_call):

return CallFunction(aten.relu, computation_call)

def sigmoid_fusion(computation_call):

return CallFunction(aten.sigmoid, computation_call)

def tanh_fusion(computation_call):

return CallFunction(aten.tanh, computation_call)

def combined_fusion(computation_call):

return CallFunction([aten.relu, aten.sigmoid, aten.tanh], computation_call)

Then figure out which one you hit with the match arg to the lowering.

This should allow you to have fewer patterns throughout the stack.

yes, it is simplified.

…onv_unary)" cc jgong5 mingfeima sanchitintel ashokei jingxu10 soumith voznesenskym penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

XiaobingSuper · 2023-04-11T02:58:29Z

@jansel @desertfire, please help review this code again. Thanks!

…onv_unary)" cc jgong5 mingfeima sanchitintel ashokei jingxu10 soumith voznesenskym penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

jansel · 2023-04-11T21:01:24Z

torch/_inductor/fx_passes/mkldnn_fusion.py

+        for pattern, computation_op in zip(_hardtanh_patterns, computation_ops):
+            _register_hardtanh_fusion_lowering(pattern, computation_op)
+
+    def _mkldnn_fusion_init():


Suggested change

def _mkldnn_fusion_init():

@functools.lru_cache(None)

def _mkldnn_fusion_init():

So this code only runs once.

jansel · 2023-04-11T21:03:48Z

torch/_inductor/fx_passes/post_grad.py

+        if torch._C.has_mkldnn:
+            from .mkldnn_fusion import _mkldnn_fusion_init
+
+            _mkldnn_fusion_init()


Move this code into a lazy_init() function with an lru_cache(None) decorator so it only runs once.

Moved the code to lazy_init(). Thanks!

…onv_unary)" cc jgong5 mingfeima sanchitintel ashokei jingxu10 soumith voznesenskym penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

XiaobingSuper · 2023-04-12T02:39:09Z

@pytorchbot merge

pytorchmergebot · 2023-04-12T02:41:05Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-04-12T02:41:10Z

Merge failed

Reason: Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x ae89899bc14a745357805942f587ada8a22ad2ee returned non-zero exit code 1

Auto-merging torch/_inductor/compile_fx.py
Auto-merging torch/_inductor/mkldnn.py
Auto-merging torch/_inductor/pattern_matcher.py
CONFLICT (content): Merge conflict in torch/_inductor/pattern_matcher.py
Auto-merging torch/_meta_registrations.py
error: could not apply ae89899bc14... inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".

Details for Dev Infra team

Raised by workflow job

XiaobingSuper · 2023-04-12T05:50:56Z

@pytorchbot merge

pytorchmergebot · 2023-04-12T05:52:49Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…97007) Pull Request resolved: #97007 Approved by: https://github.com/jgong5, https://github.com/EikanWang, https://github.com/jansel

XiaobingSuper added 3 commits March 17, 2023 03:06

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

b130ee6

[ghstack-poisoned]

Update on "inductor: rewrite mkldnn fx fusion using pattern_matcher(c…

332db2c

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

Update on "inductor: rewrite mkldnn fx fusion using pattern_matcher(c…

07cfb27

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

github-actions bot added ciflow/inductor module: inductor labels Mar 17, 2023

XiaobingSuper added a commit that referenced this pull request Mar 17, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

25254fe

ghstack-source-id: 779777fad1d385b53a45524b12c3401f787277d5 Pull Request resolved: #97007

XiaobingSuper added the release notes: inductor label Mar 17, 2023

pytorchbot added the open source label Mar 17, 2023

XiaobingSuper added a commit that referenced this pull request Mar 17, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

ddb197d

ghstack-source-id: ae74b1f32376fe8941ea8c8e3e01e3f0f45bd470 Pull Request resolved: #97007

XiaobingSuper marked this pull request as draft March 17, 2023 07:35

XiaobingSuper requested review from jgong5 and EikanWang March 17, 2023 07:35

XiaobingSuper added a commit that referenced this pull request Mar 17, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

1c1fcb3

ghstack-source-id: e5f15609e22d0171b3945f0f27aff7039170815e Pull Request resolved: #97007

Update on "inductor: rewrite mkldnn fx fusion using pattern_matcher(c…

78b1294

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

XiaobingSuper added a commit that referenced this pull request Mar 20, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

2b3c6a8

ghstack-source-id: b6ab144c346a478f87e8f7b77cf35ea55ea05500 Pull Request resolved: #97007

Update on "inductor: rewrite mkldnn fx fusion using pattern_matcher(c…

747e654

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

XiaobingSuper added a commit that referenced this pull request Mar 20, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary)

41c21ec

ghstack-source-id: 9484d4056c62d3cd1cc3c0e501c4410977f589ee Pull Request resolved: #97007

jgong5 approved these changes Mar 20, 2023

View reviewed changes

torch/_inductor/pattern_matcher.py Outdated Show resolved Hide resolved

EikanWang reviewed Mar 20, 2023

View reviewed changes

Update on "inductor: rewrite mkldnn fx fusion using pattern_matcher(c…

5ac3594

…onv_unary)" cc soumith voznesenskym penguinwu anijain2305 EikanWang jgong5 Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

This was referenced Mar 20, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(linear_unary) #97139

Closed

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_transpose_unary) #97140

Closed

github-actions bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Mar 20, 2023

XiaobingSuper mentioned this pull request Mar 20, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(binary) #97141

Closed

XiaobingSuper added 2 commits March 30, 2023 05:31

XiaobingSuper added 2 commits March 30, 2023 21:28

XiaobingSuper requested a review from desertfire April 4, 2023 07:32

XiaobingSuper mentioned this pull request Apr 6, 2023

[torch.compile] Cannot view a tensor with shape #98436

Closed

EikanWang approved these changes Apr 6, 2023

View reviewed changes

desertfire reviewed Apr 7, 2023

View reviewed changes

jansel reviewed Apr 7, 2023

View reviewed changes

XiaobingSuper added 2 commits April 10, 2023 06:27

XiaobingSuper requested review from desertfire and jansel April 10, 2023 14:28

jansel requested changes Apr 11, 2023

View reviewed changes

XiaobingSuper requested a review from jansel April 11, 2023 23:51

XiaobingSuper added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 11, 2023

jansel approved these changes Apr 12, 2023

View reviewed changes

pytorchmergebot added the merging label Apr 12, 2023

pytorchmergebot added Merged and removed merging labels Apr 12, 2023

pytorchmergebot closed this in c214c50 Apr 12, 2023

ZainRizvi pushed a commit that referenced this pull request Apr 19, 2023

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) (#…

51b6244

…97007) Pull Request resolved: #97007 Approved by: https://github.com/jgong5, https://github.com/EikanWang, https://github.com/jansel

facebook-github-bot deleted the gh/XiaobingSuper/81/head branch June 8, 2023 15:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) #97007

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) #97007

XiaobingSuper commented Mar 17, 2023 •

edited

pytorch-bot bot commented Mar 17, 2023 •

edited

jgong5 left a comment

EikanWang Mar 20, 2023

EikanWang Mar 20, 2023

XiaobingSuper Apr 4, 2023

pytorchmergebot commented Mar 30, 2023

desertfire Apr 7, 2023

jansel Apr 7, 2023

XiaobingSuper Apr 10, 2023

jansel Apr 7, 2023

XiaobingSuper Apr 11, 2023

XiaobingSuper commented Apr 11, 2023

jansel Apr 11, 2023

XiaobingSuper Apr 11, 2023

jansel Apr 11, 2023

XiaobingSuper Apr 11, 2023

XiaobingSuper commented Apr 12, 2023

pytorchmergebot commented Apr 12, 2023

pytorchmergebot commented Apr 12, 2023

XiaobingSuper commented Apr 12, 2023

pytorchmergebot commented Apr 12, 2023

		@@ -608,6 +608,279 @@ def addmm(match, mat1, mat2, inp):
		return L[aten.add](inp, L[aten.mm](mat1, mat2))


		# TODO(XiaobingSuper): move it to fx_passes/mkldnn_fuse.py

	def _mkldnn_fusion_init():
	@functools.lru_cache(None)
	def _mkldnn_fusion_init():

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) #97007

inductor: rewrite mkldnn fx fusion using pattern_matcher(conv_unary) #97007

Conversation

XiaobingSuper commented Mar 17, 2023 • edited

pytorch-bot bot commented Mar 17, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/97007

✅ No Failures

jgong5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pytorchmergebot commented Mar 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaobingSuper commented Apr 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaobingSuper commented Apr 12, 2023

pytorchmergebot commented Apr 12, 2023

Merge started

pytorchmergebot commented Apr 12, 2023

Merge failed

XiaobingSuper commented Apr 12, 2023

pytorchmergebot commented Apr 12, 2023

Merge started

XiaobingSuper commented Mar 17, 2023 •

edited

pytorch-bot bot commented Mar 17, 2023 •

edited