Inductor cpp wrapper: support QLinear #112378

chunyuan-w · 2023-10-30T09:36:54Z

Stack from ghstack (oldest at bottom):

Align the type of post_op_args in the schema of onednn::qlinear_pointwise to be the same as other fusion OPs like qconv, conv, conv_transpose, linear by changing from float[] to Scalar?[]:

pytorch/aten/src/ATen/native/quantized/library.cpp

Lines 260 to 266 in cb942ef

    
           // Conv1D/2D/3D with unary postop 
        
           m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv1d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv2d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv3d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor")); 
        
           // Conv2D with binary postop 
        
           m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv2d_pointwise.binary(Tensor qx, float x_scale, int x_zero_point, Tensor qaccum, float accum_scale, int accum_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor"));

pytorch/aten/src/ATen/native/mkldnn/RegisterMkldnnOpContextClass.cpp

Lines 48 to 59 in cb942ef

    
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_linear_pointwise(Tensor X, Tensor W, Tensor? B, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_linear_pointwise.binary(Tensor X, Tensor other, Tensor W, Tensor? B, str attr) -> Tensor Y")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_convolution_pointwise(Tensor X, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_convolution_pointwise.binary(Tensor X, Tensor other, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor Y")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_convolution_pointwise_.binary(Tensor(a!) other, Tensor X, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor(a!) Y")); 
        
           m.def(TORCH_SELECTIVE_SCHEMA( 
        
               "mkldnn::_convolution_transpose_pointwise(Tensor X, Tensor W, Tensor? B, int[] padding, int[] output_padding, int[] stride, int[] dilation, int groups, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y"));

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

[ghstack-poisoned]

pytorch-bot · 2023-10-30T09:36:59Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/112378

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c2f716e with merge base bbd5b93 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 voznesenskym penguinwu EikanWang Guobing-Chen zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

desertfire · 2023-10-31T14:22:38Z

torch/_inductor/ir.py

+            cpp_kernel="onednn::qlinear_pointwise",
+        )
+        self.cpp_kernel_key = "qlinear_pointwise"
+        self.cpp_op_schema = """


The same comment as in #112373

Same as for #112373, let me submit a subsequent PR to clean this up. Created an issue to track it: #112552

chunyuan-w · 2023-11-01T06:19:20Z

@pytorchbot merge

pytorchmergebot · 2023-11-01T06:21:31Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Based on the `Argument types` section in this [file](https://github.com/pytorch/pytorch/tree/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native#func), for non-inplace `Tensor` type in schema, it should be mapped to C++ argument of type `const Tensor&`. For `quantized_max_pool1d` and `quantized_max_pool2d`, the type of the `qx` input is `Tensor` type in the schema, thus modified the C++ type to be `const Tensor&`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L222-L223 Pull Request resolved: #112379 Approved by: https://github.com/jgong5, https://github.com/jansel ghstack dependencies: #112373, #112378

Align the type of `post_op_args` in the schema of `onednn::qlinear_pointwise` to be the same as other fusion OPs like qconv, conv, conv_transpose, linear by changing from `float[]` to `Scalar?[]`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L260-L266 https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/mkldnn/RegisterMkldnnOpContextClass.cpp#L48-L59 Pull Request resolved: pytorch#112378 Approved by: https://github.com/jgong5, https://github.com/desertfire ghstack dependencies: pytorch#112373

Based on the `Argument types` section in this [file](https://github.com/pytorch/pytorch/tree/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native#func), for non-inplace `Tensor` type in schema, it should be mapped to C++ argument of type `const Tensor&`. For `quantized_max_pool1d` and `quantized_max_pool2d`, the type of the `qx` input is `Tensor` type in the schema, thus modified the C++ type to be `const Tensor&`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L222-L223 Pull Request resolved: pytorch#112379 Approved by: https://github.com/jgong5, https://github.com/jansel ghstack dependencies: pytorch#112373, pytorch#112378

Align the type of `post_op_args` in the schema of `onednn::qlinear_pointwise` to be the same as other fusion OPs like qconv, conv, conv_transpose, linear by changing from `float[]` to `Scalar?[]`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L260-L266 https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/mkldnn/RegisterMkldnnOpContextClass.cpp#L48-L59 Pull Request resolved: pytorch#112378 Approved by: https://github.com/jgong5, https://github.com/desertfire ghstack dependencies: pytorch#112373

Based on the `Argument types` section in this [file](https://github.com/pytorch/pytorch/tree/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native#func), for non-inplace `Tensor` type in schema, it should be mapped to C++ argument of type `const Tensor&`. For `quantized_max_pool1d` and `quantized_max_pool2d`, the type of the `qx` input is `Tensor` type in the schema, thus modified the C++ type to be `const Tensor&`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L222-L223 Pull Request resolved: pytorch#112379 Approved by: https://github.com/jgong5, https://github.com/jansel ghstack dependencies: pytorch#112373, pytorch#112378

Align the type of `post_op_args` in the schema of `onednn::qlinear_pointwise` to be the same as other fusion OPs like qconv, conv, conv_transpose, linear by changing from `float[]` to `Scalar?[]`: https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/quantized/library.cpp#L260-L266 https://github.com/pytorch/pytorch/blob/cb942ef2b12134bfaa1727295380fe00ebb537c0/aten/src/ATen/native/mkldnn/RegisterMkldnnOpContextClass.cpp#L48-L59 Pull Request resolved: pytorch#112378 Approved by: https://github.com/jgong5, https://github.com/desertfire ghstack dependencies: pytorch#112373

Inductor cpp wrapper: support QLinear

63861bf

[ghstack-poisoned]

chunyuan-w requested review from jerryzh168, salilsdesai, kimishpatel, digantdesai and jianyuh as code owners October 30, 2023 09:36

pytorch-bot bot added the release notes: quantization release notes category label Oct 30, 2023

chunyuan-w mentioned this pull request Oct 30, 2023

Inductor cpp wrapper: support QConv #112373

Closed

github-actions bot added module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor ciflow/inductor labels Oct 30, 2023

chunyuan-w mentioned this pull request Oct 30, 2023

Inductor cpp wrapper: fix QMaxPool #112379

Closed

pytorchbot added the open source label Oct 30, 2023

chunyuan-w requested a review from jgong5 October 31, 2023 05:39

jgong5 approved these changes Oct 31, 2023

View reviewed changes

desertfire approved these changes Oct 31, 2023

View reviewed changes

chunyuan-w added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 1, 2023

pytorchmergebot added the merging label Nov 1, 2023

pytorchmergebot added the Merged label Nov 1, 2023

pytorchmergebot removed the merging label Nov 1, 2023

pytorchmergebot closed this in 29f3d39 Nov 1, 2023

facebook-github-bot deleted the gh/chunyuan-w/3/head branch November 4, 2023 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inductor cpp wrapper: support QLinear #112378

Inductor cpp wrapper: support QLinear #112378

chunyuan-w commented Oct 30, 2023 •

edited

pytorch-bot bot commented Oct 30, 2023 •

edited

desertfire Oct 31, 2023

chunyuan-w Nov 1, 2023

chunyuan-w commented Nov 1, 2023

pytorchmergebot commented Nov 1, 2023

	// Conv1D/2D/3D with unary postop
	m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv1d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor"));
	m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv2d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor"));
	m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv3d_pointwise(Tensor qx, float x_scale, int x_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str attr, Scalar?[] scalars, str? algorithm) -> Tensor"));

	// Conv2D with binary postop
	m.def(TORCH_SELECTIVE_SCHEMA("onednn::qconv2d_pointwise.binary(Tensor qx, float x_scale, int x_zero_point, Tensor qaccum, float accum_scale, int accum_zero_point, Tensor qw, Tensor w_scale, Tensor w_zero_point, Tensor? bias, int[] stride, int[] padding, int[] dilation, int groups, float inv_output_scale, int output_zero_point, bool fp32_output, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor"));

	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_linear_pointwise(Tensor X, Tensor W, Tensor? B, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y"));
	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_linear_pointwise.binary(Tensor X, Tensor other, Tensor W, Tensor? B, str attr) -> Tensor Y"));
	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_convolution_pointwise(Tensor X, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y"));
	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_convolution_pointwise.binary(Tensor X, Tensor other, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor Y"));
	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_convolution_pointwise_.binary(Tensor(a!) other, Tensor X, Tensor W, Tensor? B, int[] padding, int[] stride, int[] dilation, int groups, str binary_attr, Scalar? alpha, str? unary_attr, Scalar?[] unary_scalars, str? unary_algorithm) -> Tensor(a!) Y"));
	m.def(TORCH_SELECTIVE_SCHEMA(
	"mkldnn::_convolution_transpose_pointwise(Tensor X, Tensor W, Tensor? B, int[] padding, int[] output_padding, int[] stride, int[] dilation, int groups, str attr, Scalar?[] scalars, str? algorithm) -> Tensor Y"));

Inductor cpp wrapper: support QLinear #112378

Inductor cpp wrapper: support QLinear #112378

Conversation

chunyuan-w commented Oct 30, 2023 • edited

pytorch-bot bot commented Oct 30, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/112378

✅ No Failures

desertfire Oct 31, 2023

Choose a reason for hiding this comment

chunyuan-w Nov 1, 2023

Choose a reason for hiding this comment

chunyuan-w commented Nov 1, 2023

pytorchmergebot commented Nov 1, 2023

Merge started

chunyuan-w commented Oct 30, 2023 •

edited

pytorch-bot bot commented Oct 30, 2023 •

edited