[MPS] Add glu #79866

qqaatw · 2022-06-20T14:30:54Z

Adds mps op for aten::glu.out.

facebook-github-bot · 2022-06-20T14:31:04Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/79866
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours

❌ 2 New Failures

As of commit 1820dd4 (more details on the Dr. CI page):

Expand to see more

2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages

trunk / macos-11-py3-x86-64 / test (default, 2, 2, macos-12) (1/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-06-30T14:05:13.6945710Z FAIL [0.003s]: tes...duction_with_dim_cpu_bool (__main__.TestCommonCPU)

2022-06-30T14:05:13.6720750Z   test_tags_view_as_real_cpu_complex64 (__main__.TestTagsCPU) ... ok (0.002s)
2022-06-30T14:05:13.6756670Z   test_tags_view_cpu_float32 (__main__.TestTagsCPU) ... ok (0.003s)
2022-06-30T14:05:13.6782560Z   test_tags_vsplit_cpu_float32 (__main__.TestTagsCPU) ... ok (0.003s)
2022-06-30T14:05:13.6806450Z   test_tags_vstack_cpu_float32 (__main__.TestTagsCPU) ... ok (0.002s)
2022-06-30T14:05:13.6841830Z   test_tags_where_cpu_float32 (__main__.TestTagsCPU) ... ok (0.003s)
2022-06-30T14:05:13.6885620Z   test_tags_xlogy_cpu_float32 (__main__.TestTagsCPU) ... ok (0.004s)
2022-06-30T14:05:13.6912450Z   test_tags_zero__cpu_float32 (__main__.TestTagsCPU) ... ok (0.003s)
2022-06-30T14:05:13.6943960Z   test_tags_zeros_like_cpu_float32 (__main__.TestTagsCPU) ... ok (0.003s)
2022-06-30T14:05:13.6944810Z 
2022-06-30T14:05:13.6945200Z ======================================================================
2022-06-30T14:05:13.6945710Z FAIL [0.003s]: test_non_standard_bool_values_max_reduction_with_dim_cpu_bool (__main__.TestCommonCPU)
2022-06-30T14:05:13.6946510Z ----------------------------------------------------------------------
2022-06-30T14:05:13.6946810Z Traceback (most recent call last):
2022-06-30T14:05:13.6947410Z   File "/Users/runner/miniconda3/envs/build/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 377, in instantiated_test
2022-06-30T14:05:13.6947780Z     result = test(self, **param_kwargs)
2022-06-30T14:05:13.6948370Z   File "/Users/runner/miniconda3/envs/build/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 786, in test_wrapper
2022-06-30T14:05:13.6949490Z     return test(*args, **kwargs)
2022-06-30T14:05:13.6949760Z   File "test_ops.py", line 1191, in test_non_standard_bool_values
2022-06-30T14:05:13.6950020Z     self.assertEqual(expect, actual)
2022-06-30T14:05:13.6950580Z   File "/Users/runner/miniconda3/envs/build/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2237, in assertEqual
2022-06-30T14:05:13.6950900Z     assert_equal(

pull / linux-focal-py3.7-gcc7 / test (backwards_compat, 1, 1, linux.2xlarge) (2/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-06-20T15:05:11.0375858Z The PR is introduc...m to confirm whether this change is wanted or not.

2022-06-20T15:05:11.0360397Z processing existing schema:  duration_ns(__torch__.torch.classes.profiling.InstructionStats _0) -> int _0
2022-06-20T15:05:11.0361991Z processing existing schema:  source(__torch__.torch.classes.profiling.SourceStats _0) -> __torch__.torch.classes.profiling.SourceRef _0
2022-06-20T15:05:11.0364134Z processing existing schema:  line_map(__torch__.torch.classes.profiling.SourceStats _0) -> Dict(int, __torch__.torch.classes.profiling.InstructionStats) _0
2022-06-20T15:05:11.0365596Z processing existing schema:  __init__(__torch__.torch.classes.profiling._ScriptProfile _0) -> NoneType _0
2022-06-20T15:05:11.0366941Z processing existing schema:  enable(__torch__.torch.classes.profiling._ScriptProfile _0) -> NoneType _0
2022-06-20T15:05:11.0368498Z processing existing schema:  disable(__torch__.torch.classes.profiling._ScriptProfile _0) -> NoneType _0
2022-06-20T15:05:11.0371116Z processing existing schema:  _dump_stats(__torch__.torch.classes.profiling._ScriptProfile _0) -> __torch__.torch.classes.profiling.SourceStats[] _0
2022-06-20T15:05:11.0372184Z processing existing schema:  __init__(__torch__.torch.classes.c10d.ProcessGroup _0, int _1, int _2) -> NoneType _0
2022-06-20T15:05:11.0373794Z processing existing schema:  __init__(__torch__.torch.classes.c10d.Work _0) -> NoneType _0
2022-06-20T15:05:11.0375391Z processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> NoneType _0
2022-06-20T15:05:11.0375858Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2022-06-20T15:05:11.0375894Z 
2022-06-20T15:05:11.0376044Z Broken ops: [
2022-06-20T15:05:11.0376882Z 	aten::_native_decoder_only_multi_head_attention(Tensor query, Tensor key, Tensor value, int embed_dim, int num_head, Tensor qkv_weight, Tensor qkv_bias, Tensor proj_weight, Tensor proj_bias, Tensor? mask=None, Tensor? incr_key=None, Tensor? incr_value=None, bool need_weights=True, bool average_attn_weights=True) -> (Tensor, Tensor, Tensor, Tensor)
2022-06-20T15:05:11.0377704Z 	aten::_transformer_decoder_only_layer_fwd(Tensor src, int embed_dim, int num_heads, Tensor qkv_weight, Tensor qkv_bias, Tensor proj_weight, Tensor proj_bias, bool use_gelu, bool norm_first, float eps, Tensor norm_weight_1, Tensor norm_bias_1, Tensor norm_weight_2, Tensor norm_bias_2, Tensor ffn_weight_1, Tensor ffn_bias_1, Tensor ffn_weight_2, Tensor ffn_bias_2, Tensor? mask=None, Tensor? incr_key=None, Tensor? incr_value=None) -> (Tensor, Tensor, Tensor)
2022-06-20T15:05:11.0377987Z 	aten::_linalg_eigh(Tensor A, str UPLO="L", bool compute_v=True) -> (Tensor eigenvalues, Tensor eigenvectors)
2022-06-20T15:05:11.0378368Z 	aten::_linalg_eigh.eigenvalues(Tensor A, str UPLO="L", bool compute_v=True, *, Tensor(a!) eigenvalues, Tensor(b!) eigenvectors) -> (Tensor(a!) eigenvalues, Tensor(b!) eigenvectors)
2022-06-20T15:05:11.0378438Z ]
2022-06-20T15:05:11.2001852Z + cleanup
2022-06-20T15:05:11.2002040Z + retcode=1
2022-06-20T15:05:11.2002211Z + set +x

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

kulinseth

This looks good, thanks.

qqaatw · 2022-06-29T08:09:27Z

@albanD should we add ciflow/trunk label?

kulinseth · 2022-06-30T00:25:55Z

@albanD should we add ciflow/trunk label?

Done. Good point.

albanD

All looks good!

albanD · 2022-06-30T08:57:00Z

@pytorchbot merge -f

pytorchmergebot · 2022-06-30T08:58:38Z

@pytorchbot successfully started a merge job. Check the current status here

github-actions · 2022-06-30T08:59:15Z

Hey @qqaatw.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Adds mps op for `aten::glu.out`. Pull Request resolved: #79866 Approved by: https://github.com/kulinseth, https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/c980fc3d3c8db5bff6fd10dd746dc56c55b82fcd Reviewed By: seemethere Differential Revision: D37558999 fbshipit-source-id: dcfe5800d53f2da0cc2ddd3076541ae078291d23

Adds mps op for `aten::glu.out`. Pull Request resolved: pytorch#79866 Approved by: https://github.com/kulinseth, https://github.com/albanD

qqaatw · 2022-07-15T06:32:18Z

@pytorchbot label "release notes: mps" "topic: new feature"

Adds mps op for `aten::glu.out`. Pull Request resolved: pytorch#79866 Approved by: https://github.com/kulinseth, https://github.com/albanD

* MPS: Fixes (#78930) Cast integer to float in UnaryOps Add tensor dtype in key generation Enable FP16 scalars and use placeholder for alpha tensor in add/sum ops Fixes #ISSUE_NUMBER Pull Request resolved: #78930 Approved by: https://github.com/albanD * MPS: Binary cast fix by proper type promotion and remove spurious copy warning (#79185) Fixes #78019, #78020 Fixes #79185 Pull Request resolved: #79185 Approved by: https://github.com/albanD, https://github.com/razarmehr * MPS: add exponential op (#79188) Add exponential distribution Fixes #ISSUE_NUMBER Pull Request resolved: #79188 Approved by: https://github.com/razarmehr, https://github.com/albanD * [MPS] Delete unused vars from OperationUtils.mm Pull Request resolved: #79514 Approved by: https://github.com/kulinseth, https://github.com/albanD * [MPS] Fix getDefaultGenerator and copy_kernel_mps Returning reference to stack memory is really bad Pull Request resolved: #79515 Approved by: https://github.com/albanD * [MPS][BE]Do not use `new/delete[]` in `chainViewOperation` `std::array` will do just fine Pull Request resolved: #79516 Approved by: https://github.com/albanD * [MPS] Support stride of stride Fixes #79181 Pull Request resolved: #79521 Approved by: https://github.com/kulinseth * MPS: TopK raise an error if K>16 (#79677) * Error out in TopK when k>16. * Add a test case too. Fixes #78915 Pull Request resolved: #79677 Approved by: https://github.com/albanD * [MPS]: Add fix for squeezed input axes handling in BCE loss (#79676) Fixes #79527 Pull Request resolved: #79676 Approved by: https://github.com/razarmehr, https://github.com/albanD * MPS: Add amax and amin Ops with tests (#79682) * Add amax and amin with tests Fixes #ISSUE_NUMBER Pull Request resolved: #79682 Approved by: https://github.com/albanD * [MPS] Fix torch.uint8 support (#80049) `ScalarType.Byte` should be cast to `MPSDataTypeUInt8` And support for `torch.int8` as well as test those conversions in `TestMPS.test_to` Fixes #80006 Pull Request resolved: #80049 Approved by: https://github.com/albanD * [MPS] Fix binary ops between int32 tensor with int64 scalar (#80220) For some reason, tensor *op* scalar does not follow the normal binary promotion rules So cast output tensor to expected type if needed It seems that one should have casted input tensors to expected output tensor type, but it does not really work for boolean binary ops, so... Add output tensor type/shape to cached graph key Extend `TestMPS. test_add_scalars` to test for this regression Fixes #79835 Pull Request resolved: #80220 Approved by: https://github.com/albanD * [MPS] Add equal operator (#80195) Which is, in essence is composite of `eq`->`all`->`item` `native/mps/operators/Equal.cpp` is an almost verbatim copy of `native/cuda/Equal.cpp` Fix codegen by generating MPSFunctions headers Pull Request resolved: #80195 Approved by: https://github.com/albanD * [MPS] add `aten::normal.Tensor_float` `aten::normal.float_Tensor` `aten::normal.Tensor_Tensor` (#80297) Fixes #ISSUE_NUMBER Pull Request resolved: #80297 Approved by: https://github.com/albanD, https://github.com/kulinseth * [MPS] Add flip (#80214) Fixes #ISSUE_NUMBER Pull Request resolved: #80214 Approved by: https://github.com/DenisVieriu97, https://github.com/albanD * [MPS] Add logical ops (#80216) This PR adds `logical_not`, `logical_and`, `logical_or`, `logical_xor`. Pull Request resolved: #80216 Approved by: https://github.com/albanD, https://github.com/kulinseth * [MPS] Add glu (#79866) Adds mps op for `aten::glu.out`. Pull Request resolved: #79866 Approved by: https://github.com/kulinseth, https://github.com/albanD * [MPS] Fix std/var cache issue (#80502) Use `getTensorsStringKey` which has tensor shape info added as part of the key to prevent cache lookup issue when the shape of input tensor is changed. Fixes #80499 Pull Request resolved: #80502 Approved by: https://github.com/malfet, https://github.com/kulinseth * Add scatter support for view operations (#79939) * Add scatter support for view operations; #78074, #78886, #79672 * Update test_slicing_replace_column to properly test different sizes * Handle in-place changes for binary ops; add new testcase * Add new view ops testing scatter; add MPSDebugConfig.h config file for debugging purposes * Merge gatherViewTensor and scatterViewTensor into a generic function * Add scatter on demand in scatterViewOperation instead of caching it into a generic graph * Create separate graphs for scatter and gather; * Create scatter graph at scatter time Fixes #ISSUE_NUMBER Pull Request resolved: #79939 Approved by: https://github.com/razarmehr * MPS: Fix handling of 1D tensors in linear backward (#80759) Fixes ##79784 Pull Request resolved: #80759 Approved by: https://github.com/ezyang * [MPS] Move the View ops to a separate file and reduce the number of graphs created (#80491) This is dependent on the PR to go in first: #79939 Remove the data_ptr from the View Graph key which reduces the number of graphs created significantly. Don't wait when copying from MPS to MPS tensors Pull Request resolved: #80491 Approved by: https://github.com/malfet * [MPS] Add softplus backward (#79873) Fixes #ISSUE_NUMBER Pull Request resolved: #79873 Approved by: https://github.com/malfet * [MPS] Add argmin (#80828) This PR 1. adds argmin 2. refactors `reduction_type` in `ReduceOps.mm` with enum. Co-authored by Kulin Seth <kulinseth@gmail.com> Pull Request resolved: #80828 Approved by: https://github.com/malfet * [MPS] Fix LSTM batch_first output transposed (#80597) The output of LSTM with `batch_first` should be transposed back to batch first format. Fixes #80306 Pull Request resolved: #80597 Approved by: https://github.com/kulinseth * [MPS][BE] Introduce MPSUnaryCachedGraph (#81033) I.e. CachedGraph that has input and output tensors Also, add `MPSGraphCache::LookUpAs` template, which combines LookUp with static_cast to target type Pull Request resolved: #81033 Approved by: https://github.com/kulinseth * [MPS] Add test consistency from OpInfo based tests from PR 78504 (#79532) Pull Request resolved: #79532 Approved by: https://github.com/albanD, https://github.com/malfet * [MPS] Add huber loss (#80163) Fixes #ISSUE_NUMBER Pull Request resolved: #80163 Approved by: https://github.com/kulinseth, https://github.com/malfet * Remove two tests dependent on the MPS serialization checkin. * Fix lint error (FLAKE8) F401 * Remove the serialization test from test_mps as its support is not there in 1.12.1. Co-authored-by: Kulin Seth <kulinseth@gmail.com> Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com> Co-authored-by: Kulin Seth <kulin_seth@apple.com> Co-authored-by: Abhishek Pathak <abhipathak97@gmail.com> Co-authored-by: Nikita Shulga <nshulga@fb.com> Co-authored-by: qqaatw <qqaatw@gmail.com> Co-authored-by: Ramin Azarmehr <razarmehr@apple.com>

add glu mps

1820dd4

qqaatw requested review from kulinseth and bdhirsh as code owners June 20, 2022 14:30

facebook-github-bot added the cla signed label Jun 20, 2022

pytorchbot added the open source label Jun 20, 2022

qqaatw mentioned this pull request Jun 20, 2022

General MPS op coverage tracking issue #77764

Open

ngimel added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 21, 2022

kulinseth approved these changes Jun 29, 2022

View reviewed changes

kulinseth added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 30, 2022

albanD approved these changes Jun 30, 2022

View reviewed changes

pytorchmergebot added the Merged label Jun 30, 2022

pytorchmergebot closed this in c980fc3 Jun 30, 2022

kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Jul 9, 2022

[MPS] Add glu (pytorch#79866)

65de939

Adds mps op for `aten::glu.out`. Pull Request resolved: pytorch#79866 Approved by: https://github.com/kulinseth, https://github.com/albanD

pytorch-bot bot added release notes: mps Release notes category topic: new features topic category labels Jul 15, 2022

atalman pushed a commit to atalman/pytorch that referenced this pull request Jul 22, 2022

[MPS] Add glu (pytorch#79866)

a8ae7eb

Adds mps op for `aten::glu.out`. Pull Request resolved: pytorch#79866 Approved by: https://github.com/kulinseth, https://github.com/albanD

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MPS] Add glu #79866

[MPS] Add glu #79866

qqaatw commented Jun 20, 2022

facebook-github-bot commented Jun 20, 2022 •

edited

Loading

🕵️ 2 new failures recognized by patterns

trunk / macos-11-py3-x86-64 / test (default, 2, 2, macos-12) (1/2)

pull / linux-focal-py3.7-gcc7 / test (backwards_compat, 1, 1, linux.2xlarge) (2/2)

kulinseth left a comment

qqaatw commented Jun 29, 2022

kulinseth commented Jun 30, 2022

albanD left a comment

albanD commented Jun 30, 2022

pytorchmergebot commented Jun 30, 2022

github-actions bot commented Jun 30, 2022

qqaatw commented Jul 15, 2022

[MPS] Add glu #79866

[MPS] Add glu #79866

Conversation

qqaatw commented Jun 20, 2022

facebook-github-bot commented Jun 20, 2022 • edited Loading

🔗 Helpful links

❌ 2 New Failures

🕵️ 2 new failures recognized by patterns

trunk / macos-11-py3-x86-64 / test (default, 2, 2, macos-12) (1/2)

pull / linux-focal-py3.7-gcc7 / test (backwards_compat, 1, 1, linux.2xlarge) (2/2)

kulinseth left a comment

Choose a reason for hiding this comment

qqaatw commented Jun 29, 2022

kulinseth commented Jun 30, 2022

albanD left a comment

Choose a reason for hiding this comment

albanD commented Jun 30, 2022

pytorchmergebot commented Jun 30, 2022

github-actions bot commented Jun 30, 2022

qqaatw commented Jul 15, 2022

facebook-github-bot commented Jun 20, 2022 •

edited

Loading