Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nn.Linear: dispatch to bsr_dense_mm for half and bfloat16 #94825

Closed
wants to merge 27 commits into from
Closed

nn.Linear: dispatch to bsr_dense_mm for half and bfloat16 #94825

wants to merge 27 commits into from

Conversation

nikitaved
Copy link
Collaborator

@nikitaved nikitaved commented Feb 14, 2023 •
edited

@nikitaved nikitaved mentioned this pull request Feb 14, 2023
Closed
@pytorch-bot
Copy link

pytorch-bot bot commented Feb 14, 2023 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94825

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit bc7ed5c:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: sparse release notes category label Feb 14, 2023
nikitaved added a commit that referenced this pull request Feb 14, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
1c30674 
ghstack-source-id: 6790e06cf0ddf3b33831ebba8cc638f8956f631a
Pull Request resolved: #94825
@nikitaved nikitaved added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 14, 2023
nikitaved added a commit that referenced this pull request Feb 14, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
99395f7 
ghstack-source-id: 133a63c1ceea38d9d24e8e3d5d8622ed139284a1
Pull Request resolved: #94825
nikitaved added a commit that referenced this pull request Feb 15, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
463feb3 
ghstack-source-id: bf2abe355707ed10c0fd913a002a3a8cbb7f2e9f
Pull Request resolved: #94825
@nikitaved nikitaved mentioned this pull request Feb 15, 2023
Closed
@nikitaved nikitaved added the module: sparse Related to torch.sparse label Feb 15, 2023
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
96ae529 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
9f22de0 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
9272076 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
f14766f 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
e06462c 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
adee82d 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
01b7989 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
aa474cb 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
4875767 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Copy link
Collaborator Author

@cpuhrsch , let's try this one as well?

@cpuhrsch
Copy link
Contributor

cpuhrsch commented Mar 7, 2023

@nikitaved - Were you able to locally reproduce the deadlock with CUDA 11.6?

@nikitaved
Copy link
Collaborator Author

nikitaved commented Mar 7, 2023 •
edited

No, I was not. Looks like the issue might have been fixed on the Triton's side. The first PR in the stack does run the tests though with no issues so far, so we could give it some more grace time?

@cpuhrsch
Copy link
Contributor

cpuhrsch commented Mar 7, 2023

@nikitaved - you can install a different (older) version of Triton via pypi and PyTorch will pick up on it.

For example

pip install triton==2.0.0.dev20221202

If you do this with a build based on 11.6, does it reproduce the issue? Is there any way we can reconstruct the environment under which you've observed these failures? I'm worried to land something this critical that we know has had issues in the past without resolving them.

@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
61d6e16 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
nikitaved added a commit that referenced this pull request Apr 3, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
b75f23e 
ghstack-source-id: 90a98a5b496a6da02080d1fae976f1c3d7de26bd
Pull Request resolved: #94825
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
c4aa47a 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
nikitaved added a commit that referenced this pull request Apr 4, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
b16d7b2 
ghstack-source-id: f8650b35ad68727fccafa65a8c3ba87f1f98ad2a
Pull Request resolved: #94825
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
5502b19 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
nikitaved added a commit that referenced this pull request Apr 4, 2023
@nikitaved
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16
3d5300d 
ghstack-source-id: 68f44145c6df2794fb557826389a0d94db132182
Pull Request resolved: #94825
@nikitaved
Copy link
Collaborator Author

@cpuhrsch , I was not able to fail it, so, I guess, this one should be good to go.

@nikitaved nikitaved mentioned this pull request Apr 5, 2023
Closed
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
6101854 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
d45a198 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
ce75dec 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Copy link
Collaborator Author

@cpuhrsch , @albanD , could you please stamp this one?

cpuhrsch
cpuhrsch approved these changes Apr 13, 2023
View reviewed changes
Copy link
Contributor

@cpuhrsch cpuhrsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing. Triton in eager mode!

@cpuhrsch
Copy link
Contributor

Unfortunately there's still a lint error left, but otherwise let's give this another go.

@nikitaved
Update on "nn.Linear: dispatch to bsr_dense_mm for half and bfloat16"
bc7ed5c 
cc alexsamardzic pearu cpuhrsch amjames bhosmer

[ghstack-poisoned]
@nikitaved
Copy link
Collaborator Author

@pytorchbot merge -g

@pytorch-bot
Copy link

pytorch-bot bot commented Apr 14, 2023

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: -g

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci} ...

Try @pytorchbot --help for more info.

@nikitaved
Copy link
Collaborator Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

ZainRizvi pushed a commit that referenced this pull request Apr 19, 2023
@nikitaved @ZainRizvi
nn.Linear: dispatch to bsr_dense_mm for half and bfloat16 (#94825)
e1858e6 
Pull Request resolved: #94825
Approved by: https://github.com/albanD, https://github.com/cpuhrsch
@xwang233 xwang233 mentioned this pull request May 31, 2023
Closed
pytorchmergebot pushed a commit that referenced this pull request Jun 1, 2023
@xwang233 @pytorchmergebot
Skip test test_triton_bsr_dense_bmm if not TEST_WITH_TORCHINDUCTOR [v…
6340aa5 
…2] (#102660)

Test was originally skipped in #98462

Not sure why it was removed in #94825

Now the test hits CUDA illegal memory access on H100 again after #101163

Pull Request resolved: #102660
Approved by: https://github.com/zou3519
alimoezzi pushed a commit to alimoezzi/pytorch that referenced this pull request Jun 3, 2023
@xwang233 @alimoezzi
Skip test test_triton_bsr_dense_bmm if not TEST_WITH_TORCHINDUCTOR [v…
98b0db6 
…2] (pytorch#102660)

Test was originally skipped in pytorch#98462

Not sure why it was removed in pytorch#94825

Now the test hits CUDA illegal memory access on H100 again after pytorch#101163

Pull Request resolved: pytorch#102660
Approved by: https://github.com/zou3519
@facebook-github-bot facebook-github-bot deleted the gh/nikitaved/21/head branch June 8, 2023 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdhirsh bdhirsh bdhirsh left review comments

@cpuhrsch cpuhrsch cpuhrsch approved these changes

@malfet malfet Awaiting requested review from malfet

@albanD albanD Awaiting requested review from albanD

@amjames amjames Awaiting requested review from amjames

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged merging module: sparse Related to torch.sparse open source release notes: sparse release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@nikitaved @cpuhrsch @albanD @pytorchmergebot @kit1980 @amjames @bdhirsh @pytorchbot