Multi-GPU inference and Specify which GPUs to be used during inference

[EthanC111]

I have two questions:

1. I attempted multi-GPU inference (8 GPU inference on A100) on Llama-13B. I followed the steps described in [https://github.com/CUDA error: out of memory #188], first running $ ray start --head and then llm = LLM(model=<your model>, tensor_parallel_size=8).
   However, I got the following error:
   (Worker pid=1027546) AssertionError: 32001 is not divisible by 8 [repeated 7x across cluster]
   Is there any way to resolve this issue?

2. Additionally, is there a way to specify which GPUs are used during inference? I tried using os.environ["CUDA_VISIBLE_DEVICES"]="2" but it doesn't seem to work - it continues to use the first GPU.

Thanks!

[xcxhy]

@EthanC111 the second error, you can use CUDA_VISIBLE_DEVICES=2 python test.py like this code on terminal to instead of setting in code.

[bigmover]

I use "CUDA_VISIABLE_DEVICES=1 python -m vllm.entrypoints.api_server " to place server on device 1. But it doesn't work! What should I do to figure it out?

[JackYangzg]

I found this cannot special using which GPU

[zhuohan123]

For the first error, what exactly was the model you use? Did you change the vocabulary size? If your vocabulary size cannot be divided by the number of GPUs, you will see this error.

[alex-ji-repo]

how do we set vocabulary size? is it in the sampling_params?

[renfeier]

I attempted multi-GPU inference (8 GPU inference on A100) on Llama-13B. I first running $ ray start --head and then llm = LLM(model="./models/open_llama_13b", tensor_parallel_size=4)
and then i got the flowing output:

2023-07-04 16:57:32,881 INFO worker.py:1452 -- Connecting to existing Ray cluster at address: 30.152.83.176:6379...
[2023-07-04 16:57:32,888 I 186137 186137] global_state_accessor.cc:356: This node has an IP address of 30.152.83.253, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
2023-07-04 16:57:32,901 INFO worker.py:1636 -- Connected to Ray cluster.

it blocked , no more log output
@zhuohan123 what's wrong with me?