Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue]: #3060

Closed
hbfreed opened this issue Apr 23, 2024 · 2 comments
Closed

[Issue]: #3060

hbfreed opened this issue Apr 23, 2024 · 2 comments

Comments

@hbfreed
Copy link

hbfreed commented Apr 23, 2024

Problem Description

After installing 6.1, when I use torch.compile, I get the following traceback (I can put the whole thing in a pastebin, it was just long):
If I don't compile, the train script runs just fine. Happy to provide any more details to get things sorted out.

File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Internal Triton PTX codegen error:
ptxas fatal : Value 'sm_110' is not defined for option 'gpu-name'

Operating System

22.04.4 LTS (Jammy Jellyfish)

CPU

EPYC 7402P

GPU

AMD Radeon RX 7900 XTX

ROCm Version

ROCm 6.0.0

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

@hbfreed
Copy link
Author

hbfreed commented Apr 23, 2024

Update: just downgraded to 6.0.2, it did not fix this problem. I've tried both regular triton and the triton nightly, for context there.

@hbfreed
Copy link
Author

hbfreed commented Apr 24, 2024

I fixed this, I just fully reinstalled torch and triton, making sure to uninstall the triton that torch packages with it.

@hbfreed hbfreed closed this as completed Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant