[Bug]: With --cpu-offload-gb, deepseek-moe-16b-chat got different response, even if the temperature is zero. #16731
Closed as not planned
Description
Your current environment
The output of `python collect_env.py`
vllm/vllm-openai latest 24d76f8822cb
🐛 Describe the bug
As the title, using --cpu-offload-gb results in different answer. And, the answer is not related to the query.
Inference.py
from openai import OpenAI
import requests
APIURL = "http://localhost:8626/v1"
client = OpenAI(
# This is the default and can be omitted
api_key="EMPTY",
base_url=APIURL,
timeout=3600
)
model = requests.get(url=APIURL + "/models").json()["data"][0]["id"]
messages = [{"role": "user", "content": "Where could I get the best Italian food in town?"}]
completion = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=150,
temperature=0,
)
print(completion.choices[0].message.content)docker without --cpu-offload-gb
docker run --gpus all -it --rm --ipc=host \
--ulimit memlock=-1 --ulimit stack=10000000000 \
-v /mnt/my_models/deepseek-moe-16b-chat:/model -p 8626:8000 \
--name=inference-vllm vllm/vllm-openai:latest \
--max-model-len 4096 --tensor-parallel-size 2 --served-model-name inference-vllm --trust-remote-code \
--gpu-memory-utilization=0.95 --model /model
# run inference.py got:
# I’m!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!docker with --cpu-offload-gb 1
docker run --gpus all -it --rm --ipc=host \
--ulimit memlock=-1 --ulimit stack=10000000000 \
-v /mnt/my_models/deepseek-moe-16b-chat:/model -p 8626:8000 \
--name=inference-vllm vllm/vllm-openai:latest \
--max-model-len 4096 --tensor-parallel-size 2 --served-model-name inference-vllm --trust-remote-code \
--gpu-memory-utilization=0.95 --model /model --cpu-offload-gb 1
# run inference.py got:
# There are many great Italian restaurants in town, and the best one for you will depend on your personal preferences. Some popular Italian restaurants in town include:
#
# Il Fornaio: This restaurant offers a variety of Italian dishes, including pasta, pizza, and seafood.
# Trattoria Romana: This restaurant is known for its authentic Roman-style pizza and pasta dishes.
# Pizzeria Mozza: This restaurant is famous for its wood-fired pizzas and other Italian-inspired dishes.
# Osteria Mozza: This restaurant is another popular spot for Italian food, with a focus on fresh, seasonal ingredients.
# Eataly: This restaurant offers a wide variety of Italian dishes, including pasta, pizza,Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.