-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AMD ROCm docker images support (+ optimization) #94
Comments
yes I love this, I have an amd device and happy to try with it |
Awesome, love the proactivity!!! let me create a draft PR. Do you have ROCm installed and can use pytorch with ROCM? @hiepxanh |
Here would be some instructions on how to get in installed. In my opinion, it should run out of the box with rocm, the question would be to build and run a docker + the performance. |
I have ROCm and AMD rx 6600 card, but I get a lot of issue while testing, pytorch not support windows, the unbutu image eat 20gbs for just only ROCm, I think we not ready now. I see vulkan work great with llama.cpp, maybe that is a good option to run the model. Let's keep this issue open while I'm watching AMD team. |
Sadly, Ubuntu/Linux (no WSL) and a error free installation of rocm is a strict requirement for ROCm. |
Yes, it correct, the WSL and docker is great place to install, I have failed last time. I will do futher test once I have freetime |
I am running Ubuntu 22.10, with a Navi 23 [Radeon RX 6650 XT] and ROCm drivers installed (rocminfo and clinfo). I'll give infinity a shot on this system. When I run --help, I do not see rocm listed as a device, and when I run infinity with no special options, it picks "cpu" as a device. How should I run? |
I can report Infinity works perfectly well using ROCm accelerated PytTorch on a 7900XTX. Just one tip, if you're not using the MI250X and MI300X series like me, set this variable before starting Infinity, to avoid PyTorch errors complaining about no HIPBLAS support: TORCH_BLAS_PREFER_HIPBLASLT=0 |
Did you build infinity-emb from scratch using a different pytorch that the one in pyproject.toml? Personally I am using the pre-built docker container image michaelf34/infinity:latest. @hvico if you have a custom build, I'd like to know your recipe! |
Hi. I didn't, I just installed the latest pip wheel, and then installed the official nightly Python ROCm packages (replacing the torch distribution by that one). To dockerize this I froze that virtualenv, started from the officlal ROCm Pytorch docker image, and added the proper pip install -r requirements.txt from the exported file. I am not sharing that file because it has many other dependencies unrelated with infinity, so it makes no sense to use that as a template. But this is the main procedure I followed. Hope it helps. |
I am planning to evaluate hardware agnostic options
torch.compile()
/float16
etc for AMDThe text was updated successfully, but these errors were encountered: