Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error running evaluation on dual GPUs #10

Open
valine opened this issue Apr 17, 2023 · 3 comments
Open

Error running evaluation on dual GPUs #10

valine opened this issue Apr 17, 2023 · 3 comments

Comments

@valine
Copy link

valine commented Apr 17, 2023

I'm attempting to run the model on dual RTX 4090s. Enabling this would be a great update and would allow more people to run the full float16 model.

Some changes would need to be made, starting by passing kwargs = { "device_map": "auto", "max_memory": {i: "13GiB" for i in range(num_gpus)}, } to LlamaForCausalLM.from_pretrained.

After making this change the model loads but throws the following error when submitting text:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Full Logs
minigpt4-error-logs.txt

@matsuren
Copy link

I successfully run on dual RTX3090s.
Try with CUDA_VISIBLE_DEVICES=0 python demo.py --cfg-path eval_configs/minigpt4_eval.yaml.

@valine
Copy link
Author

valine commented Apr 18, 2023

I successfully run on dual RTX3090s.
Try with CUDA_VISIBLE_DEVICES=0 python demo.py --cfg-path eval_configs/minigpt4_eval.yaml.

Curious, what's your VRAM usage between the cards? With CUDA_VISIBLE_DEVICES=0 I can run the model in 8bit model but my second card is idle. I'm hoping to run the full 16bit model split between my two cards.

@matsuren
Copy link

One is 16GB and the other is no usage. CUDA_VISIBLE_DEVICES=0 just limit visibility from python program. e.g., CUDA_VISIBLE_DEVICES=0: only GPU_id = 0 is visible, CUDA_VISIBLE_DEVICES=1: only GPU_id = 1 is visible, and CUDA_VISIBLE_DEVICES=0,1: both GPU_id=0 and 1 are visible.

If you want to run the inference using two cards then you need to do something like this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants