-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrading to VLLM 0.4.1 - TypeError #38
Comments
Looks like VLLM changed a parameter name. Here's my fix to get this project working with VLLM 0.4.1
|
I'm not a git expert, but attempted to create a PR for you |
@ProVega thanks for the finding! We need to update the same file in multiple folders. I will create another pr to fix this later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I recently upgraded to VLLM 0.4.1 and now get the following error. This looks internal to bento not my service (which is basically the default Llama 3 sample). here is my requirements.txt
accelerate==0.29.3 bentoml>=1.2.12 packaging==24.0 torch==2.2.1 transformers==4.40.0 vllm==0.4.1
2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Initializing service error
Traceback (most recent call last):
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_sdk/service/factory.py", line 230, in call
instance = self.inner()
File "/home/admin/BentoVLLM/llama3-8b-instruct/bentovllm_openai/utils.py", line 77, in init
self.openai_serving_completion = OpenAIServingCompletion(
TypeError: OpenAIServingCompletion.init() got an unexpected keyword argument 'served_model'
2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Traceback (most recent call last):
File "/home/admin/.local/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/home/admin/.local/lib/python3.10/site-packages/bentoml/_internal/server/base_app.py", line 74, in lifespan
on_startup()
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 313, in create_instance
self._service_instance = self.service()
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_sdk/service/factory.py", line 230, in call
instance = self.inner()
File "/home/admin/BentoVLLM/llama3-8b-instruct/bentovllm_openai/utils.py", line 77, in init
self.openai_serving_completion = OpenAIServingCompletion(
TypeError: OpenAIServingCompletion.init() got an unexpected keyword argument 'served_model'
2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Application startup failed. Exiting.
The text was updated successfully, but these errors were encountered: