Skip to content

[ROCm] set hipblas workspace #138791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 7 commits into from
Closed

[ROCm] set hipblas workspace #138791

wants to merge 7 commits into from

Conversation

jeffdaily
Copy link
Collaborator

@jeffdaily jeffdaily commented Oct 24, 2024 •
edited by pytorch-bot bot
Loading

Fixes #138532.

This brings hipblas behavior in line with cublas behavior with respect to setting the workspace to an allocation from the caching allocator as well as the env var HIPBLAS_WORKSPACE_CONFIG.

cc @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

@jeffdaily
[ROCm] set hipblas workspace
79c3031 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect
to setting the workspace to an allocation from the caching allocator as
well as the env var HIPBLAS_WORKSPACE_CONFIG.
@jeffdaily jeffdaily added module: rocm AMD GPU support for Pytorch release notes: rocm mandatorylabel rocm This tag is for PRs from ROCm team rocm priority high priority ROCm PRs from performance or other aspects ciflow/rocm Trigger "default" config CI on ROCm labels Oct 24, 2024
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 24, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138791

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8f09561 with merge base 8aedc64 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

naromero77amd
naromero77amd reviewed Oct 24, 2024
View reviewed changes
naromero77amd
naromero77amd approved these changes Oct 24, 2024
View reviewed changes
Copy link
Collaborator

@naromero77amd naromero77amd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that this fix on an MI-300.

@jeffdaily jeffdaily added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 25, 2024
@soulitzer soulitzer requested review from malfet and ngimel October 28, 2024 15:03
@soulitzer soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 28, 2024
@naromero77amd
Copy link
Collaborator

We have some urgency in getting this upstream since its a show stopper bug for some users. Pinging a couple of upstream maintainers @malfet @eqy.

I have already reviewed and approved it.

malfet
malfet approved these changes Oct 28, 2024
View reviewed changes
Copy link
Contributor

@malfet malfet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please consider formatting changes

jeffdaily and others added 2 commits October 28, 2024 14:40
@jeffdaily @malfet
Update aten/src/ATen/cuda/CublasHandlePool.cpp
bf3b2f4 
nit: missed lint

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
@jeffdaily
update hip.rst with latest rocBLAS workspace default sizes
8f09561
@jeffdaily
Copy link
Collaborator Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@hliuca hliuca mentioned this pull request Oct 28, 2024
Closed
rahulsingh-intel pushed a commit to rahulsingh-intel/pytorch that referenced this pull request Oct 29, 2024
@jeffdaily @malfet @rahulsingh-intel
[ROCm] set hipblas workspace (pytorch#138791)
a7d4fae 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect to setting the workspace to an allocation from the caching allocator as well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy, https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
rahulsingh-intel pushed a commit to rahulsingh-intel/pytorch that referenced this pull request Nov 5, 2024
@jeffdaily @malfet @rahulsingh-intel
[ROCm] set hipblas workspace (pytorch#138791)
ded6bd2 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect to setting the workspace to an allocation from the caching allocator as well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy, https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
jeffdaily added a commit to ROCm/pytorch that referenced this pull request Nov 14, 2024
@jeffdaily @malfet
[ROCm] set hipblas workspace (pytorch#138791)
5ac3d1b 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect to setting the workspace to an allocation from the caching allocator as well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy, https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
@jeffdaily jeffdaily mentioned this pull request Nov 14, 2024
Merged
jeffdaily added a commit to ROCm/pytorch that referenced this pull request Nov 14, 2024
@jeffdaily @malfet
[ROCm] set hipblas workspace (pytorch#138791)
a42f3ac 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect to setting the workspace to an allocation from the caching allocator as well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy, https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
@jeffdaily jeffdaily mentioned this pull request Nov 14, 2024
Merged
pruthvistony pushed a commit to ROCm/pytorch that referenced this pull request Nov 15, 2024
@jeffdaily @malfet
[release/2.4] set hipblas workspace (pytorch#138791) (#1716)
b8dd07d 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect
to setting the workspace to an allocation from the caching allocator as
well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy,
https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
pruthvistony pushed a commit to ROCm/pytorch that referenced this pull request Nov 15, 2024
@jeffdaily @malfet
[release/2.5] set hipblas workspace (pytorch#138791) (#1715)
17dc494 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect
to setting the workspace to an allocation from the caching allocator as
well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy,
https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
@jeffdaily jeffdaily mentioned this pull request Feb 3, 2025
Closed
jithunnair-amd pushed a commit to ROCm/pytorch that referenced this pull request Mar 17, 2025
@jeffdaily @malfet @jithunnair-amd
[release/2.4] set hipblas workspace (pytorch#138791) (#1716)
e8e52b2 
Fixes pytorch#138532.

This brings hipblas behavior in line with cublas behavior with respect
to setting the workspace to an allocation from the caching allocator as
well as the env var HIPBLAS_WORKSPACE_CONFIG.

Pull Request resolved: pytorch#138791
Approved by: https://github.com/naromero77amd, https://github.com/eqy,
https://github.com/malfet

Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eqy eqy eqy approved these changes

@malfet malfet malfet approved these changes

@syed-ahmed syed-ahmed Awaiting requested review from syed-ahmed syed-ahmed is a code owner

@jithunnair-amd jithunnair-amd Awaiting requested review from jithunnair-amd jithunnair-amd is a code owner

@ngimel ngimel Awaiting requested review from ngimel

+1 more reviewer

@naromero77amd naromero77amd naromero77amd approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/rocm Trigger "default" config CI on ROCm ciflow/trunk Trigger trunk jobs on your pull request Merged module: rocm AMD GPU support for Pytorch open source release notes: rocm mandatorylabel rocm priority high priority ROCm PRs from performance or other aspects rocm This tag is for PRs from ROCm team triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[ROCm] MI300X Tunable ops causes 100GB of Memory Leak leading to OOM

7 participants

@jeffdaily @naromero77amd @pytorchmergebot @eqy @malfet @soulitzer @pytorchbot