Your current environment
Colab notebooks, A100
🐛 Describe the bug
I have no idea what's wrong. This model works with normal pipeline, but fails with vllm. It was saved to 16 bit and built off Unsloth/Llama3.18b Instruct
This works ->
from transformers import pipeline
prompt = (
"Do aliens suffer?"
)
gen = pipeline("text-generation", model=best_model, tokenizer=tokenizer)
print(gen(prompt, max_new_tokens=256, do_sample=True, temperature=0.7)[0]["generated_text"])
This fails-> It does not give me a root cause this error is all I see after it hangs for about 3 minutes
INFO 05-03 23:45:17 [config.py:717] This model supports multiple tasks: {'generate', 'reward', 'classify', 'score', 'embed'}. Defaulting to 'generate'.
INFO 05-03 23:45:17 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 05-03 23:45:19 [utils.py:2382] We must use the spawn multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing for more information. Reason: CUDA is initialized
RuntimeError Traceback (most recent call last)
in <cell line: 0>()
1 from vllm import LLM, SamplingParams
----> 2 llm=LLM(model="full_model", tokenizer= "full_model", gpu_memory_utilization=0.9)
3 prompts = ["Do aliens suffer?"] # Your question
4
5 # Generate responses
8 frames
/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py in _wait_for_engine_startup(self)
428 if len(events) > 1 or events[0][0] != sync_input_socket:
429 # One of the core processes exited.
--> 430 raise RuntimeError("Engine core initialization failed. "
431 "See root cause above.")
432
RuntimeError: Engine core initialization failed. See root cause above.
Before submitting a new issue...
Your current environment
Colab notebooks, A100
🐛 Describe the bug
I have no idea what's wrong. This model works with normal pipeline, but fails with vllm. It was saved to 16 bit and built off Unsloth/Llama3.18b Instruct
This works ->
from transformers import pipeline
prompt = (
"Do aliens suffer?"
)
gen = pipeline("text-generation", model=best_model, tokenizer=tokenizer)
print(gen(prompt, max_new_tokens=256, do_sample=True, temperature=0.7)[0]["generated_text"])
This fails-> It does not give me a root cause this error is all I see after it hangs for about 3 minutes
INFO 05-03 23:45:17 [config.py:717] This model supports multiple tasks: {'generate', 'reward', 'classify', 'score', 'embed'}. Defaulting to 'generate'.
INFO 05-03 23:45:17 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 05-03 23:45:19 [utils.py:2382] We must use the
spawnmultiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing for more information. Reason: CUDA is initializedRuntimeError Traceback (most recent call last)
in <cell line: 0>()
1 from vllm import LLM, SamplingParams
----> 2 llm=LLM(model="full_model", tokenizer= "full_model", gpu_memory_utilization=0.9)
3 prompts = ["Do aliens suffer?"] # Your question
4
5 # Generate responses
8 frames
/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py in _wait_for_engine_startup(self)
428 if len(events) > 1 or events[0][0] != sync_input_socket:
429 # One of the core processes exited.
--> 430 raise RuntimeError("Engine core initialization failed. "
431 "See root cause above.")
432
RuntimeError: Engine core initialization failed. See root cause above.
Before submitting a new issue...