[PyTorch] MHA: simplify gemm_nt #72460

swolchok · 2022-02-07T21:30:57Z

Stack from ghstack:

Just call existing matmul (which, IIUC, handles batching itself) rather than doing a few view ops. (Please let me know if this is actually a bad idea and why!)

Differential Revision: D33961843

Just call existing matmul (which, IIUC, handles batching itself) rather than doing a few view ops. (Please let me know if this is actually a bad idea and why!) Differential Revision: [D33961843](https://our.internmc.facebook.com/intern/diff/D33961843/) [ghstack-poisoned]

pytorch-bot · 2022-02-07T21:31:02Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/82d7005f23a9cab14212e8a5d42125b9bd946252/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
linux-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-manywheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
linux-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/trunk`, `ciflow/xla`	✅ triggered
linux-bionic-rocm4.5-py3.7	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/rocm`, `ciflow/trunk`	✅ triggered
linux-docs	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/docs`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-vulkan-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
windows-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
docker-builds	`ciflow/all`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`, `ciflow/trunk`	🚫 skipped
linux-docs-push	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-11-py3-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.7-gcc7-debug	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
periodic-win-vs2019-cuda11.5-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped

facebook-github-bot · 2022-02-07T21:31:07Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/72460
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 76fe7c3 (more details on the Dr. CI page):

2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

linux-bionic-py3.7-clang9 / test (xla, 1, 1, linux.2xlarge) (1/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-02-14T18:04:53.9345980Z /var/lib/jenkins/w... at::Tensor&, const at::Tensor&, c10::string_view)

2022-02-14T18:04:53.9340304Z  at::Tensor XLANativeFunctions::gelu(const at::Tensor& self) {
2022-02-14T18:04:53.9340632Z             ^~~~~~~~~~~~~~~~~~
2022-02-14T18:04:53.9341007Z In file included from /var/lib/jenkins/workspace/xla/torch_xla/csrc/aten_xla_type.cpp:13:0:
2022-02-14T18:04:53.9341685Z /var/lib/jenkins/workspace/xla/torch_xla/csrc/XLANativeFunctions.h:182:19: error: candidate is: static at::Tensor torch_xla::XLANativeFunctions::gelu(const at::Tensor&, c10::string_view)
2022-02-14T18:04:53.9342322Z  static at::Tensor gelu(const at::Tensor & self, c10::string_view approximate);
2022-02-14T18:04:53.9342653Z                    ^~~~
2022-02-14T18:04:53.9343688Z /var/lib/jenkins/workspace/xla/torch_xla/csrc/aten_xla_type.cpp:1514:12: error: prototype for ‘at::Tensor torch_xla::XLANativeFunctions::gelu_backward(const at::Tensor&, const at::Tensor&)’ does not match any in class ‘torch_xla::XLANativeFunctions’
2022-02-14T18:04:53.9344435Z  at::Tensor XLANativeFunctions::gelu_backward(const at::Tensor& grad,
2022-02-14T18:04:53.9344869Z             ^~~~~~~~~~~~~~~~~~
2022-02-14T18:04:53.9345266Z In file included from /var/lib/jenkins/workspace/xla/torch_xla/csrc/aten_xla_type.cpp:13:0:
2022-02-14T18:04:53.9345980Z /var/lib/jenkins/workspace/xla/torch_xla/csrc/XLANativeFunctions.h:183:19: error: candidate is: static at::Tensor torch_xla::XLANativeFunctions::gelu_backward(const at::Tensor&, const at::Tensor&, c10::string_view)
2022-02-14T18:04:53.9346713Z  static at::Tensor gelu_backward(const at::Tensor & grad_output, const at::Tensor & self, c10::string_view approximate);
2022-02-14T18:04:53.9347117Z                    ^~~~~~~~~~~~~
2022-02-14T18:04:54.2252813Z [73/179] c++ -MMD -MF /var/lib/jenkins/workspace/xla/build/temp.linux-x86_64-3.7/torch_xla/csrc/ops/convolution_backward_overrideable.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/var/lib/jenkins/workspace/xla -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-tensorflow -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-bin -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-tensorflow/external/protobuf_archive/src -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-tensorflow/external/com_google_protobuf/src -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-tensorflow/external/eigen_archive -I/var/lib/jenkins/workspace/xla/third_party/tensorflow/bazel-tensorflow/external/com_google_absl -I/var/lib/jenkins/workspace -I/var/lib/jenkins/workspace/torch/csrc -I/var/lib/jenkins/workspace/torch/lib/tmp_install/include -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/opt/conda/include/python3.7m -c -c /var/lib/jenkins/workspace/xla/torch_xla/csrc/ops/convolution_backward_overrideable.cpp -o /var/lib/jenkins/workspace/xla/build/temp.linux-x86_64-3.7/torch_xla/csrc/ops/convolution_backward_overrideable.o -std=c++14 -Wno-sign-compare -Wno-deprecated-declarations -Wno-return-type -DNDEBUG -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_clang"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1002"' -DTORCH_EXTENSION_NAME=_XLAC -D_GLIBCXX_USE_CXX11_ABI=1
2022-02-14T18:04:54.2257353Z cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
2022-02-14T18:04:54.2257923Z In file included from /var/lib/jenkins/workspace/c10/util/Logging.h:28:0,
2022-02-14T18:04:54.2258413Z                  from /var/lib/jenkins/workspace/c10/core/TensorImpl.h:14,
2022-02-14T18:04:54.2259229Z                  from /opt/conda/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:21,
2022-02-14T18:04:54.2259935Z                  from /opt/conda/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
2022-02-14T18:04:54.2260464Z                  from /var/lib/jenkins/workspace/torch/csrc/lazy/core/hash.h:12,
2022-02-14T18:04:54.2260949Z                  from /var/lib/jenkins/workspace/xla/torch_xla/csrc/ir.h:19,

linux-xenial-py3.7-clang7-onnx / test (default, 1, 2, linux.2xlarge) (2/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-02-14T18:00:48.1478075Z �[31mERROR: pip's ... the source of the following dependency conflicts.

2022-02-14T18:00:46.4484412Z ++ stat --format %U /opt/conda/bin/pip
2022-02-14T18:00:46.4524811Z + PIP_USER=jenkins
2022-02-14T18:00:46.4527533Z ++ id -u -n
2022-02-14T18:00:46.4586417Z + CURRENT_USER=jenkins
2022-02-14T18:00:46.4586699Z + [[ jenkins = root ]]
2022-02-14T18:00:46.4587016Z + pip -q uninstall -y hypothesis
2022-02-14T18:00:46.8935455Z + pip -q uninstall -y coverage
2022-02-14T18:00:47.2053935Z �[33mWARNING: Skipping coverage as it is not installed.�[0m
2022-02-14T18:00:47.2631124Z + pip -q install attrs==18.1.0 -f https://s3.amazonaws.com/ossci-linux/wheels/attrs-18.1.0-py2.py3-none-any.whl
2022-02-14T18:00:47.6226646Z �[33mWARNING: Skipping page https://s3.amazonaws.com/ossci-linux/wheels/attrs-18.1.0-py2.py3-none-any.whl because the HEAD request got Content-Type: binary/octet-stream.The only supported Content-Type is text/html�[0m
2022-02-14T18:00:48.1478075Z �[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2022-02-14T18:00:48.1478891Z pytest 7.0.1 requires attrs>=19.2.0, but you have attrs 18.1.0 which is incompatible.�[0m
2022-02-14T18:00:48.2300589Z + pip -q install coverage==4.5.1 -f https://s3.amazonaws.com/ossci-linux/wheels/coverage-4.5.1-cp36-cp36m-macosx_10_12_x86_64.whl
2022-02-14T18:00:48.5949738Z �[33mWARNING: Skipping page https://s3.amazonaws.com/ossci-linux/wheels/coverage-4.5.1-cp36-cp36m-macosx_10_12_x86_64.whl because the HEAD request got Content-Type: binary/octet-stream.The only supported Content-Type is text/html�[0m
2022-02-14T18:00:49.6421213Z + pip -q install hypothesis==3.44.6 -f https://s3.amazonaws.com/ossci-linux/wheels/hypothesis-3.44.6-py3-none-any.whl
2022-02-14T18:00:49.9907122Z �[33mWARNING: Skipping page https://s3.amazonaws.com/ossci-linux/wheels/hypothesis-3.44.6-py3-none-any.whl because the HEAD request got Content-Type: binary/octet-stream.The only supported Content-Type is text/html�[0m
2022-02-14T18:00:51.3102754Z + EXTRA_TESTS=()
2022-02-14T18:00:51.3103385Z + [[ linux-xenial-py3.7-clang7-onnx == *-cuda* ]]
2022-02-14T18:00:51.3103935Z + [[ linux-xenial-py3.7-clang7-onnx == *-rocm* ]]
2022-02-14T18:00:51.3104301Z + rocm_ignore_test=()
2022-02-14T18:00:51.3104705Z + [[ linux-xenial-py3.7-clang7-onnx == *-rocm* ]]

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

Just call existing matmul (which, IIUC, handles batching itself) rather than doing a few view ops. (Please let me know if this is actually a bad idea and why!) Differential Revision: [D33961843](https://our.internmc.facebook.com/intern/diff/D33961843/) [ghstack-poisoned]

Summary: Pull Request resolved: #72460 Just call existing matmul (which, IIUC, handles batching itself) rather than doing a few view ops. (Please let me know if this is actually a bad idea and why!) ghstack-source-id: 149067333 Test Plan: CI Reviewed By: ngimel Differential Revision: D33961843 fbshipit-source-id: ace37ad3110e1134db6c8b638ae302f0d556e00a

Summary: Pull Request resolved: pytorch/pytorch#72460 Just call existing matmul (which, IIUC, handles batching itself) rather than doing a few view ops. (Please let me know if this is actually a bad idea and why!) ghstack-source-id: 149067333 Test Plan: CI Reviewed By: ngimel Differential Revision: D33961843 fbshipit-source-id: ace37ad3110e1134db6c8b638ae302f0d556e00a (cherry picked from commit 258231c0f951bd701da179eaedc1ef795416c53f)

pytorch-bot bot added the ciflow/default label Feb 7, 2022

facebook-github-bot added the cla signed label Feb 7, 2022

swolchok added 2 commits February 8, 2022 16:14

swolchok mentioned this pull request Feb 10, 2022

[PyTorch] Handle non-vectorizable parameters for native MHA CUDA rescale kernel #72671

Closed

pytorchmergebot closed this in 49611a3 Feb 15, 2022

facebook-github-bot deleted the gh/swolchok/448/head branch February 18, 2022 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PyTorch] MHA: simplify gemm_nt #72460

[PyTorch] MHA: simplify gemm_nt #72460

Uh oh!

swolchok commented Feb 7, 2022 •

edited

Loading

Uh oh!

pytorch-bot bot commented Feb 7, 2022

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Feb 7, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[PyTorch] MHA: simplify gemm_nt #72460

[PyTorch] MHA: simplify gemm_nt #72460

Uh oh!

Conversation

swolchok commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 7, 2022

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 2 new failures recognized by patterns

linux-bionic-py3.7-clang9 / test (xla, 1, 1, linux.2xlarge) (1/2)

linux-xenial-py3.7-clang7-onnx / test (default, 1, 2, linux.2xlarge) (2/2)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

swolchok commented Feb 7, 2022 •

edited

Loading

facebook-github-bot commented Feb 7, 2022 •

edited

Loading