[Bug]: When tensor_parallel_size>1, RuntimeError: Cannot re-initialize CUDA in forked subprocess.

[excelsimon]

Your current environment

vllm version: '0.5.0.post1'

🐛 Describe the bug

When I set tensor_parallel_size=1, it works well.
But, if I set tensor_parallel_size>1, below error occurs:
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method.
After I add

import torch
import multiprocessing
torch.multiprocessing.set_start_method('spawn')

the same RuntimeError still occurs.

[youkaichao]

please paste your full code. you might initialized cuda before using vLLM.

[yuchenlin]

I'm also having the same issue with the latest version of vllm + gemma-2-27B-it

[yuchenlin]

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

[youkaichao]

I can run the following code without any issues:

from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="google/gemma-2-27b-it", tensor_parallel_size=2)
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[excelsimon]

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

It works for me. Thank you~

[yuchenlin]

I can run the following code without any issues:

from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="google/gemma-2-27b-it", tensor_parallel_size=2)
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

i'm not sure. maybe it's about the version issue of powerinfer? and I actually find that even I used the above solution to make vllm able to generate, the quality is not as good as other Gemma-2-27B inference (both under greedy decoding).

[henry-y]

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

it also works for me! thank you!

[CharlesRiggins]

I have encountered the same issue. I solved it by making VLLM_WORKER_MULTIPROC_METHOD=spawn as mentioned by @yuchenlin
Now I'm wondering why I get this error and why setting VLLM_WORKER_MULTIPROC_METHOD solves it. Please help us know and make it clear.

[lonngxiang]

not work for me

export VLLM_WORKER_MULTIPROC_METHOD=spawn;CUDA_VISIBLE_DEVICES=1 vllm serve  /ai/minicpmv --host 192.168.2.238 --port 10868 --max-model-len 10000 --trust-remote-code --api-key token-abc123 --gpu_memory_utilization 1 --trust-remote-code 

[rin2401]

from peft import PeftModel, PeftConfig
i import peft made this error too

[youkaichao]

@rin2401 try to use distributed_executor_backend="ray" ?

[IIDCII]

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

if you're using this in a Python notebook run the following first on a reset kernel:

import os
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"