[NVFuser] don't decompose linear if we don't have shape info #75770

davidberard98 · 2022-04-13T23:56:58Z

Stack from ghstack:

-> [NVFuser] don't decompose linear if we don't have shape info #75770

$ gpui python test/test_jit_cuda_fuser.py -k test_linear
./fsx/users/dberard/pytorch/torch/testing/_internal/common_utils.py:425: UserWarning: concrete shape for linear input & weight are required to decompose into matmul + bias (T
riggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:2059.)
  return callable(*args, **kwargs)
.
----------------------------------------------------------------------
Ran 2 tests in 41.073s

OK

[ghstack-poisoned]

facebook-github-bot · 2022-04-13T23:57:04Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/75770
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
❓Need help or want to give feedback on the CI? Visit our office hours

💊 CI failures summary and remediations

As of commit 29db863 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-15T21:08:48.6817426Z The PR is introduc...m to confirm whether this change is wanted or not.

2022-04-15T21:08:48.6790574Z processing existing schema:  text(__torch__.torch.classes.profiling.SourceRef _0) -> (str _0)
2022-04-15T21:08:48.6791676Z processing existing schema:  count(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-15T21:08:48.6792963Z processing existing schema:  duration_ns(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-15T21:08:48.6794359Z processing existing schema:  source(__torch__.torch.classes.profiling.SourceStats _0) -> (__torch__.torch.classes.profiling.SourceRef _0)
2022-04-15T21:08:48.6796311Z processing existing schema:  line_map(__torch__.torch.classes.profiling.SourceStats _0) -> (Dict(int, __torch__.torch.classes.profiling.InstructionStats) _0)
2022-04-15T21:08:48.6814923Z processing existing schema:  __init__(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-15T21:08:48.6815494Z processing existing schema:  enable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-15T21:08:48.6816001Z processing existing schema:  disable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-15T21:08:48.6816771Z processing existing schema:  _dump_stats(__torch__.torch.classes.profiling._ScriptProfile _0) -> (__torch__.torch.classes.profiling.SourceStats[] _0)
2022-04-15T21:08:48.6817184Z processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (NoneType _0)
2022-04-15T21:08:48.6817426Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2022-04-15T21:08:48.6817434Z 
2022-04-15T21:08:48.6817499Z Broken ops: [
2022-04-15T21:08:48.6817787Z 	aten::_nested_from_padded(Tensor padded, Tensor cpu_nested_shape_example, bool fuse_transform_0213=False) -> (Tensor)
2022-04-15T21:08:48.6818013Z 	aten::_nested_from_padded_and_nested_example(Tensor padded, Tensor nt_example) -> (Tensor)
2022-04-15T21:08:48.6818182Z 	aten::to_padded_tensor(Tensor self, float padding) -> (Tensor)
2022-04-15T21:08:48.6818233Z ]
2022-04-15T21:08:48.7814740Z + cleanup
2022-04-15T21:08:48.7814886Z + retcode=1
2022-04-15T21:08:48.7815003Z + set +x
2022-04-15T21:08:48.7857661Z ##[error]Process completed with exit code 1.

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[ghstack-poisoned]

ghstack-source-id: 6d1c70a Pull Request resolved: #75770

jjsjann123 · 2022-04-14T07:14:40Z

torch/csrc/jit/codegen/cuda/partition.cpp

    }
    return false;
  }
+  // linear can't be fused, but for other reasons it needs to be parsible


ah... Sorry about this. We've separated fusion/profile node registration. So instead of special case on block fusion, we should have updated parser.cpp. Can't believe we haven't run into this earlier 😕

Let me push a patch to your PR.

jjsjann123 · 2022-04-14T07:18:57Z

torch/csrc/jit/codegen/cuda/graph_fuser.cpp

-    TORCH_INTERNAL_ASSERT(
-        mat0_size.has_value() && mat1_size.has_value(),
-        "concrete shape for linear input & weight are required");
+    if (!mat0_size.has_value() || !mat1_size.has_value()) {


Do you think it's ok that we throw a one time warning here? Or maybe even only throw warning when we are in debug.
The assert was put here so that we would be well aware of missing profiling information.

jjsjann123 · 2022-04-14T07:20:00Z

test/test_jit_cuda_fuser.py

+                inp = torch.rand((x, x + i)).cuda()
+                weight = torch.rand((x + 2, x + i)).cuda()
+                bias = torch.rand((x, x + 2)).cuda()
+                y += torch.sin(torch.nn.functional.linear(inp, weight, bias))


Out of curiosity, why would this case not have profiling information on linear output?

Across the two invocations of linear (i.e. i=0 and i=1) there are different shapes for the inputs of linear

we should be unrolling this loop though.. i guess the profiling happens first, then unrolling

[ghstack-poisoned]

ghstack-source-id: 6b72dab Pull Request resolved: #75770

jjsjann123 · 2022-04-15T17:52:17Z

torch/csrc/jit/codegen/cuda/parser.cpp

          nullptr);
    }

-    {


Looks like we don't need to add profiling for linear now. Wondering if some other pass were doing that. Removing this as a whole should break TestCudaFuser.test_linear but it apparently didn't :)

I'm thinking more like this one to be on the safer side. (waiting on build to test this)
https://github.com/pytorch/pytorch/pull/75897/files#diff-9738ee51d55cdf479f41b018d61ca43ed2ae10c13a20e21ec45b7578e82a23d2R2457-R2460

double checked this on my local build. ProfileRegistry is profiling all differentiable node so we don't have to register linear for profiling in nvfuser. But I still do prefer to have it registered just to be on the safe side.

Nevertheless, since this is not blocking, I'm stamping it

jjsjann123 · 2022-04-15T20:47:48Z

torch/csrc/jit/codegen/cuda/parser.cpp

          nullptr);
    }

-    {


double checked this on my local build. ProfileRegistry is profiling all differentiable node so we don't have to register linear for profiling in nvfuser. But I still do prefer to have it registered just to be on the safe side.

Nevertheless, since this is not blocking, I'm stamping it

[ghstack-poisoned]

ghstack-source-id: 83263fd Pull Request resolved: #75770

davidberard98 · 2022-04-15T20:56:46Z

Thanks for the help @jjsjann123 - tested your changes from #75897 and merged them into this commit

jjsjann123 · 2022-04-17T17:34:52Z

Is there still anything blocking this and should we ping the bot to merge it?

trying to cherry-pick upstream fixes to our local branch.

davidberard98 · 2022-04-17T17:36:31Z

@pytorchmergebot merge this

pytorchmergebot · 2022-04-17T17:37:53Z

Merge failed due to Matched rule superuser, but it was not reviewed yet by any of:esantorella,NivekT,blefaudeux,ngimel,deeptigp, ...
Raised by https://github.com/pytorch/pytorch/actions/runs/2180649374

davidberard98 · 2022-04-18T14:23:02Z

@pytorchmergebot merge this

github-actions · 2022-04-18T14:25:14Z

Hey @davidberard98.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Pull Request resolved: pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

Summary: Pull Request resolved: #75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/ebb60a8b2f495ada503b6386aef0ad61f44b7fac Reviewed By: mehtanirav Differential Revision: D35721647 Pulled By: davidberard98 fbshipit-source-id: bea898927cd6dd740f8ceb09528b3d8dcb71c4cd

Pull Request resolved: #75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta (cherry picked from commit ebb60a8)

Pull Request resolved: pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta Co-authored-by: David Berard <dberard@fb.com>

Pull Request resolved: pytorch/pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

Pull Request resolved: pytorch/pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta Co-authored-by: David Berard <dberard@fb.com>

[NVFuser] don't decompose linear if we don't have shape info

8dceeb9

[ghstack-poisoned]

davidberard98 mentioned this pull request Apr 13, 2022

[testing] -D_GLIBCXX_ASSERTIONS #75766

Closed

facebook-github-bot added the cla signed label Apr 13, 2022

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Apr 13, 2022

davidberard98 requested a review from jjsjann123 April 13, 2022 23:57

davidberard98 linked an issue Apr 13, 2022 that may be closed by this pull request

[NVFuser] Linear decomposition fails when input or weight don't have concrete shapes #75768

Closed

Update on "[NVFuser] don't decompose linear if we don't have shape info"

ab2676e

[ghstack-poisoned]

Update on "[NVFuser] don't decompose linear if we don't have shape info"

9043b91

[ghstack-poisoned]

Update on "[NVFuser] don't decompose linear if we don't have shape info"

46d6ae6

[ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Apr 14, 2022

[NVFuser] don't decompose linear if we don't have shape info

590480f

ghstack-source-id: 6d1c70a Pull Request resolved: #75770

jjsjann123 reviewed Apr 14, 2022

View reviewed changes

Update on "[NVFuser] don't decompose linear if we don't have shape info"

53f7750

[ghstack-poisoned]

Update on "[NVFuser] don't decompose linear if we don't have shape info"

4c0b8f4

[ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Apr 14, 2022

[NVFuser] don't decompose linear if we don't have shape info

076645c

ghstack-source-id: 6b72dab Pull Request resolved: #75770

davidberard98 requested a review from jjsjann123 April 15, 2022 17:05

jjsjann123 reviewed Apr 15, 2022

View reviewed changes

jjsjann123 approved these changes Apr 15, 2022

View reviewed changes

Update on "[NVFuser] don't decompose linear if we don't have shape info"

29db863

[ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Apr 15, 2022

[NVFuser] don't decompose linear if we don't have shape info

25e916d

ghstack-source-id: 83263fd Pull Request resolved: #75770

robieta self-requested a review April 18, 2022 14:21

robieta approved these changes Apr 18, 2022

View reviewed changes

pytorchmergebot closed this in ebb60a8 Apr 18, 2022

davidberard98 added the topic: not user facing topic category label Apr 18, 2022

jjsjann123 pushed a commit to csarofeen/pytorch that referenced this pull request Apr 18, 2022

[NVFuser] don't decompose linear if we don't have shape info

f8a9e32

Pull Request resolved: pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

jjsjann123 mentioned this pull request Apr 18, 2022

[NVFuser] don't decompose linear if we don't have shape info csarofeen/pytorch#1592

Merged

jjsjann123 pushed a commit to csarofeen/pytorch that referenced this pull request Apr 18, 2022

[NVFuser] don't decompose linear if we don't have shape info

7954f8c

Pull Request resolved: pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

jjsjann123 mentioned this pull request Apr 19, 2022

[NVFuser] Linear decomposition fails when input or weight don't have concrete shapes #75768

Closed

malfet pushed a commit that referenced this pull request Apr 20, 2022

[NVFuser] don't decompose linear if we don't have shape info

976eb7c

Pull Request resolved: #75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta (cherry picked from commit ebb60a8)

facebook-github-bot deleted the gh/davidberard98/92/head branch April 22, 2022 14:17

jjsjann123 pushed a commit to jjsjann123/nvfuser that referenced this pull request Oct 29, 2022

[NVFuser] don't decompose linear if we don't have shape info

ece53ee

Pull Request resolved: pytorch/pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

jjsjann123 pushed a commit to jjsjann123/nvfuser that referenced this pull request Nov 10, 2022

[NVFuser] don't decompose linear if we don't have shape info

d79ae05

Pull Request resolved: pytorch/pytorch#75770 Approved by: https://github.com/jjsjann123, https://github.com/robieta

[NVFuser] don't decompose linear if we don't have shape info #75770

[NVFuser] don't decompose linear if we don't have shape info #75770

Uh oh!

Conversation

davidberard98 commented Apr 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Apr 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Uh oh!

jjsjann123 Apr 14, 2022

Choose a reason for hiding this comment

Uh oh!

jjsjann123 Apr 14, 2022

Choose a reason for hiding this comment

Uh oh!

jjsjann123 Apr 14, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 Apr 14, 2022

Choose a reason for hiding this comment

Uh oh!

eellison May 13, 2022

Choose a reason for hiding this comment

Uh oh!

jjsjann123 Apr 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjsjann123 Apr 15, 2022

Choose a reason for hiding this comment

Uh oh!

jjsjann123 Apr 15, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 commented Apr 15, 2022

Uh oh!

jjsjann123 commented Apr 17, 2022

Uh oh!

davidberard98 commented Apr 17, 2022

Uh oh!

pytorchmergebot commented Apr 17, 2022

Uh oh!

davidberard98 commented Apr 18, 2022

Uh oh!

github-actions bot commented Apr 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

davidberard98 commented Apr 13, 2022 •

edited

Loading

facebook-github-bot commented Apr 13, 2022 •

edited

Loading

jjsjann123 Apr 15, 2022 •

edited

Loading