ValueError: The precision of the fractional quantity of resource

[sunyuhan19981208]

Description

I tried the demo code, but got an error like this:

  File "/opt/miniconda3/lib/python3.8/site-packages/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/opt/miniconda3/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/opt/miniconda3/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 87, in __init__
    worker_cls = ray.remote(
  File "/opt/miniconda3/lib/python3.8/site-packages/ray/_private/worker.py", line 2879, in _make_remote
    ray_option_utils.validate_actor_options(options, in_options=False)
  File "/opt/miniconda3/lib/python3.8/site-packages/ray/_private/ray_option_utils.py", line 308, in validate_actor_options
    actor_options[k].validate(k, v)
  File "/opt/miniconda3/lib/python3.8/site-packages/ray/_private/ray_option_utils.py", line 38, in validate
    raise ValueError(possible_error_message)
ValueError: The precision of the fractional quantity of resource node:172.17.0.8 cannot go beyond 0.0001

And my code is below:

from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="my local path", tensor_parallel_size=4)
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[WoosukKwon]

Hi, thanks for trying out vLLM! It seems the error is related to Ray. If you are not using multi GPU, could you uninstall Ray and try again?

[sunyuhan19981208]

INFO 06-21 10:37:39 llm_engine.py:59] Initializing an LLM engine with config: model='/home/sunyuhan/syh/sunyuhan/zju/opt-350m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 10:37:47 llm_engine.py:128] # GPU blocks: 8078, # CPU blocks: 2730
Processed prompts: 100%|████| 4/4 [00:00<00:00, 13.24it/s]
Prompt: 'The capital of France is', Generated text: ' Paris.</s>'
Prompt: 'Hello, my name is', Generated text: " Joel. I'm on a mission to save the people of the forest from a"
Prompt: 'The president of the United States is', Generated text: " about to be arrested.\nI'm waiting for some that take a selfie and"
Prompt: 'The future of AI is', Generated text: " creating AI with 'no brainer'\nIf you've worked in the workplace"

@WoosukKwon I can successfully infer using single GPU and a small model, but I want to use multi-GPUs to infer big model, what should I do then?

[WoosukKwon]

Oh, we omitted the instructions for multi-GPU inference. We will add it in our doc.

First, please install and run Ray:

$ pip install ray
$ ray start --head

Then, use the tensor_parallel_size argument in the LLM class:

llm = LLM(model=<your model>, tensor_parallel_size=2)  # Inference with 2 GPUs 

[sunyuhan19981208]

still not work....
Here is my ray status, the vllm script raise the same error as before.

#ray status
======== Autoscaler status: 2023-06-21 11:54:33.398400 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_88217bad91df9d797ae0fe7538f102db5c37c612b2cdfe128a9341ed
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/48.0 CPU
 0.0/8.0 GPU
 0B/114.58GiB memory
 0B/9.31GiB object_store_memory

Demands:
 (no resource demands)

[929359291]

So what should I do? I also have this problem, multiple GPUs cannot run

INFO 06-21 16:38:03 llm_engine.py:59] Initializing an LLM engine with config: model='/data/llama-30b', dtype=torch.bfloat16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=4, seed=0)
INFO 06-21 16:38:03 tokenizer_utils.py:30] Using the LLaMA fast tokenizer in 'hf-internal-testing/llama-tokenizer' to avoid potential protobuf errors.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 87, in __init__
    worker_cls = ray.remote(
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/worker.py", line 2879, in _make_remote
    ray_option_utils.validate_actor_options(options, in_options=False)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/ray_option_utils.py", line 308, in validate_actor_options
    actor_options[k].validate(k, v)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/ray_option_utils.py", line 38, in validate
    raise ValueError(possible_error_message)
ValueError: The precision of the fractional quantity of resource node:121.14.126.35 cannot go beyond 0.0001
>>> llm = LLM(model="/data/30b-apple-trans-chat", tensor_parallel_size=4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 143, in from_engine_args
    distributed_init_method, devices = initialize_cluster(parallel_config)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/ray_utils.py", line 40, in initialize_cluster
    ray.init(address=ray_address)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/worker.py", line 1478, in init
    raise RuntimeError(
RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.
>>> llm = LLM(model="/data/30b-apple-trans-chat", tensor_parallel_size=4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/llm_engine.py", line 143, in from_engine_args
    distributed_init_method, devices = initialize_cluster(parallel_config)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/vllm/engine/ray_utils.py", line 40, in initialize_cluster
    ray.init(address=ray_address)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/myenv/lib/python3.8/site-packages/ray/_private/worker.py", line 1478, in init
    raise RuntimeError(
RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

[MotzWanted]

I'm also having this issue

[AliJahan]

Same here. I followed the instruction above and still the same error when using different tensor_parallel values (2,4,6) for an 8x Tesla T4 server

[zhuohan123]

Turns out this is a new limit introduced by the latest Ray. Just fixed the error.

[AliJahan]

It works now. Thank you @zhuohan123!

[jianyu-cs]

Thanks, but how to specify which GPUs to use?

[user-ZJ]

pip install ray==2.0.0 work for me