[Usage]: How to use vllm in parallel

[bravelyi]

Your current environment

from vllm import LLM, SamplingParams

llm = LLM(
model="/Models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
tensor_parallel_size=2, # 2 张 GPU 处理张量并行
pipeline_parallel_size=2, # 2 张 GPU 处理管道并行
dtype="bfloat16",
disable_ray=True
)

采样参数

sampling_params = SamplingParams(temperature=0.7, top_p=0.9)

生成测试文本

prompts = ["你好，大语言模型是什么？"]
outputs = llm.generate(prompts, sampling_params)

输出结果

for output in outputs:
print(f"Prompt: {output.prompt}\nGenerated text: {output.outputs[0].text}")

CUDA_VISIBLE_DEVICES=2,3 python code/API.py

How would you like to use vllm

2025-03-18 11:21:32,945 INFO worker.py:1636 -- Connecting to existing Ray cluster at address: 192.168.1.216:6379...
2025-03-18 11:21:32,958 INFO worker.py:1812 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265
WARNING 03-18 11:21:32 ray_utils.py:320] The number of required GPUs exceeds the total number of available GPUs in the placement group.
INFO 03-18 11:21:42 ray_utils.py:214] Waiting for creating a placement group of specs for 10 seconds. specs=[{'node:192.168.1.216': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check ray status to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 03-18 11:22:02 ray_utils.py:214] Waiting for creating a placement group of specs for 30 seconds. specs=[{'node:192.168.1.216': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check ray status to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 03-18 11:22:42 ray_utils.py:214] Waiting for creating a placement group of specs for 70 seconds. specs=[{'node:192.168.1.216': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check ray status to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.

My vllm and cuda versions are as follows:
print("Torch version:", torch.version)
Torch version: 2.5.1+cu124

print("CUDA available:", torch.cuda.is_available())
CUDA available: True
print("CUDA version:", torch.version.cuda)
CUDA version: 12.4
print("Number of GPUs:", torch.cuda.device_count())
Number of GPUs: 4

for i in range(torch.cuda.device_count()):
... print(f"GPU {i}: {torch.cuda.get_device_name(i)}")
...
GPU 0: NVIDIA L20
GPU 1: NVIDIA L20
GPU 2: NVIDIA L20
GPU 3: NVIDIA L20
Name: vllm
Version: 0.6.6
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email:
License: Apache 2.0
Requires: aiohttp, blake3, cloudpickle, compressed-tensors, depyf, einops, fastapi, filelock, gguf, importlib_metadata, lark, lm-format-enforcer, mistral_common, msgspec, numpy, nvidia-ml-py, openai, outlines, partial-json-parser, pillow, prometheus-fastapi-instrumentator, prometheus_client, protobuf, psutil, py-cpuinfo, pydantic, pyyaml, pyzmq, ray, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, xformers, xgrammar
Required-by:

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.