-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vllm compatibility with latest release #43908
Comments
Thanks for opening this @lynkz-matt-psaltis! To clarify– are you trying to migrate Ray and RayLLM to use Pydantic v2 internally? One of the reasons that Ray (and Serve in particular) still uses the Pydantic v1 APIs is to allow users with Pydantic v1 applications to still run Serve. We have a compatibility layer that makes this possible. For the time being, we're planning to keep this structure to provide support for as many applications as possible. Is there a reason that vllm is incompatible when Ray uses the Pydantic v1 APIs internally? I'm not sure why that would affect vllm. |
Hi @shrekris-anyscale thanks for reaching out. Our goal is to be able to run latest VLLM with RayLLM. RayLLM is currently pinned to a range in the vllm 0.2.x versions however 0.3.x and now 0.4.0 has dropped today. The compatibility problems come from the interop between VLLM and Ray mixing Pydantic v1 & v2 base models. Pydantic's stance on this configuration is:
So we set about doing an internal upgrade of both Ray and RayLLM to remove the use of the Pydantic v1 compatibility layer and move fully to v2. This works when combined with a number of code and signature changes due to breaking changes in VLLM. It is my understanding that while Ray continues to use the Pydantic v1 compatibility layer, VLLM must remain pinned to 0.2.x versions. Sounds like this could get interesting - Would love to know your thoughts :) |
What failures do you see if you install Pydantic v2 and proceed to use Ray (with the compatibility layer) and vLLM?
This is surprising because we've had a number of users run Pydantic v2 with Ray successfully. The point of the compatibility layer is to enable that. I'm curious what issues are popping up. |
Description
Ray currently uses Pydantic v2's v1 compatibility layer which isn't compatible with newer versions of vllm which have fully migrated to v2. I'm currently working on aligning Ray and RayLLM with package versions that align to vllm and wanted to open a dialog around potentially contributing this work back to the community.
Use case
No response
The text was updated successfully, but these errors were encountered: