-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running grok-1 with Hugging Face int8 weights with ROCm 6.0 fails #130
Comments
The current code assumes that we use 8 GPUs, hence |
@Spurthi-Bhat-ScalersAI |
@ibab how much memory should be enough? |
Please see the steps below to install for ROCm
|
@ibab Yes, I have 8 GPUs with around ~190 GB VRAM memory, which I assume is enough to run the model? |
Yes, I am using the AMD Instinct MI300X GPU.
@rahulbatra85 Thank you for the quick support regarding the ROCm-JAX! |
@rahulbatra85 Got it working with the ROCm-JAX latest release! Thanks once again for the release and detailed steps! |
Reproducing steps:
Clone the grok-1 repo
Install the jax library:
pip install -U "jax[cuda12_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
Install the requirements:
Download the Hugging Face weights:
huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt/tensor* --local-dir checkpoints/ckpt-0 --local-dir-use-symlinks False
The local mesh config change in
run.py
fileRun the
run.py
fileThe error below:
Issues:
run.py
work with the int8 HuggingFace weights?grok-1
model on AMD ROCm 6.0?The text was updated successfully, but these errors were encountered: