[ray serve] incompatibility with pydantic >=2.0

[Yangqing]

What happened + What you expected to happen

Basically, I was trying to run the ray serve example code in the documentation with fastapi integration:

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

from ray import serve


app = FastAPI()


@serve.deployment
@serve.ingress(app)
class EchoServer:
    @app.websocket("/")
    async def echo(self, ws: WebSocket):
        await ws.accept()

        try:
            while True:
                text = await ws.receive_text()
                await ws.send_text(text)
        except WebSocketDisconnect:
            print("Client disconnected.")


serve_app = serve.run(EchoServer.bind())

expecting it would work out of box. And it produces pydantic user error as follows:

---------------------------------------------------------------------------
PydanticUserError                         Traceback (most recent call last)
Cell In[3], line 3
      1 from fastapi import FastAPI, WebSocket, WebSocketDisconnect
----> 3 from ray import serve
      6 app = FastAPI()
      9 @serve.deployment
     10 @serve.ingress(app)
     11 class EchoServer:

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/__init__.py:4
      1 import ray._private.worker
      3 try:
----> 4     from ray.serve.api import (
      5         build,
      6         deployment,
      7         get_deployment,
      8         get_replica_context,
      9         ingress,
     10         list_deployments,
     11         run,
     12         shutdown,
     13         start,
     14         delete,
     15         Application,
     16         BuiltApplication,
     17         Deployment,
     18         multiplexed,
     19         get_multiplexed_model_id,
     20     )
     21     from ray.serve.air_integrations import PredictorDeployment
     22     from ray.serve.batching import batch

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/api.py:15
     12 from ray.dag import DAGNode
     13 from ray.util.annotations import Deprecated, PublicAPI
---> 15 from ray.serve.built_application import BuiltApplication
     16 from ray.serve._private.client import ServeControllerClient
     17 from ray.serve.config import AutoscalingConfig, DeploymentConfig, HTTPOptions

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/built_application.py:7
      1 from typing import (
      2     Dict,
      3     Optional,
      4     List,
      5 )
----> 7 from ray.serve.deployment import Deployment
      8 from ray.util.annotations import PublicAPI
     11 @PublicAPI(stability="alpha")
     12 class ImmutableDeploymentDict(dict):

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/deployment.py:14
      4 from typing import (
      5     Any,
      6     Callable,
   (...)
     10     Union,
     11 )
     12 from ray._private.usage.usage_lib import TagKey, record_extra_usage_tag
---> 14 from ray.serve.context import get_global_client
     15 from ray.dag.dag_node import DAGNodeBase
     16 from ray.dag.class_node import ClassNode

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/context.py:12
     10 import ray
     11 from ray.exceptions import RayActorError
---> 12 from ray.serve._private.client import ServeControllerClient
     13 from ray.serve._private.common import ReplicaTag
     14 from ray.serve._private.constants import SERVE_CONTROLLER_NAME, SERVE_NAMESPACE

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/_private/client.py:25
     18 from ray.serve.config import DeploymentConfig, HTTPOptions
     19 from ray.serve._private.constants import (
     20     CLIENT_POLLING_INTERVAL_S,
     21     CLIENT_CHECK_CREATION_POLLING_INTERVAL_S,
     22     MAX_CACHED_HANDLES,
     23     SERVE_DEFAULT_APP_NAME,
     24 )
---> 25 from ray.serve._private.deploy_utils import get_deploy_args
     26 from ray.serve.controller import ServeController
     27 from ray.serve.exceptions import RayServeException

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/_private/deploy_utils.py:8
      5 import time
      7 from ray.serve.config import ReplicaConfig, DeploymentConfig
----> 8 from ray.serve.schema import ServeApplicationSchema
      9 from ray.serve._private.constants import SERVE_LOGGER_NAME
     10 from ray.serve._private.common import DeploymentInfo

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/schema.py:134
    124                     raise ValueError(
    125                         "runtime_envs in the Serve config support only "
    126                         "remote URIs in working_dir and py_modules. Got "
    127                         f"error when parsing URI: {e}"
    128                     )
    130         return v
    133 @PublicAPI(stability="beta")
--> 134 class DeploymentSchema(
    135     BaseModel, extra=Extra.forbid, allow_population_by_field_name=True
    136 ):
    137     """
    138     Specifies options for one deployment within a Serve application. For each deployment
    139     this can optionally be included in `ServeApplicationSchema` to override deployment
    140     options specified in code.
    141     """
    143     name: str = Field(
    144         ..., description=("Globally-unique name identifying this deployment.")
    145     )

File ~/anaconda3/lib/python3.8/site-packages/ray/serve/schema.py:242, in DeploymentSchema()
    231 ray_actor_options: RayActorOptionsSchema = Field(
    232     default=DEFAULT.VALUE, description="Options set for each replica actor."
    233 )
    235 is_driver_deployment: bool = Field(
    236     default=DEFAULT.VALUE,
    237     description="Indicate Whether the deployment is driver deployment "
    238     "Driver deployments are spawned one per node.",
    239 )
    241 @root_validator
--> 242 def num_replicas_and_autoscaling_config_mutually_exclusive(cls, values):
    243     if values.get("num_replicas", None) not in [DEFAULT.VALUE, None] and values.get(
    244         "autoscaling_config", None
    245     ) not in [DEFAULT.VALUE, None]:
    246         raise ValueError(
    247             "Manually setting num_replicas is not allowed "
    248             "when autoscaling_config is provided."
    249         )

File ~/anaconda3/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py:222, in root_validator(pre, skip_on_failure, allow_reuse, *__args)
    212 warn(
    213     'Pydantic V1 style `@root_validator` validators are deprecated.'
    214     ' You should migrate to Pydantic V2 style `@model_validator` validators,'
   (...)
    217     stacklevel=2,
    218 )
    220 if __args:
    221     # Ensure a nice error is raised if someone attempts to use the bare decorator
--> 222     return root_validator()(*__args)  # type: ignore
    224 if allow_reuse is True:  # pragma: no cover
    225     warn(_ALLOW_REUSE_WARNING_MESSAGE, DeprecationWarning)

File ~/anaconda3/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py:228, in root_validator(pre, skip_on_failure, allow_reuse, *__args)
    226 mode: Literal['before', 'after'] = 'before' if pre is True else 'after'
    227 if pre is False and skip_on_failure is not True:
--> 228     raise PydanticUserError(
    229         'If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`.'
    230         ' Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.',
    231         code='root-validator-pre-skip',
    232     )
    234 wrap = partial(_decorators_v1.make_v1_generic_root_validator, pre=pre)
    236 def dec(f: Callable[..., Any] | classmethod[Any, Any, Any] | staticmethod[Any, Any]) -> Any:

PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.

For further information visit https://errors.pydantic.dev/2.3/u/root-validator-pre-skip

I believe there is the general compatibility issue reported by others in e.g. #37372 and #37019, but just submitting this to signal the bug specifically wrt ray serve. Hope it is helpful!

Versions / Dependencies

ray: 2.6.3 (default one shipped in the anyscale platform)
pydantic: 2.3.0

Reproduction script

Do "pip install -U pydantic" to update to 2.x.

Then, run the example code at https://docs.ray.io/en/latest/serve/http-guide.html#fastapi-http-deployments

import ray
import requests
from fastapi import FastAPI
from ray import serve

app = FastAPI()


@serve.deployment(route_prefix="/hello")
@serve.ingress(app)
class MyFastAPIDeployment:
    @app.get("/")
    def root(self):
        return "Hello, world!"


serve.run(MyFastAPIDeployment.bind())
resp = requests.get("http://localhost:8000/hello")
assert resp.json() == "Hello, world!"

Issue Severity

None

[richardliaw]

Hey @Yangqing thanks a bunch for reporting! We'll take a close look.

[PythonFZ]

We are currently also struggeling with this, because parts of our code already use pydantic >=2.0.

[dioptre]

This also breaks compatibility with the dashboard too

[shrekris-anyscale]

This also breaks compatibility with the dashboard too

Could you give more details? What errors are you seeing?

[dioptre]

You can’t install dashboards if you are using pydantic >2 (I’m using poetry)

…

On Tue, Sep 26, 2023 at 17:35 shrekris-anyscale ***@***.***> wrote: This also breaks compatibility with the dashboard too Could you give more details? What errors are you seeing? — Reply to this email directly, view it on GitHub <#39722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFZTGC2U27DAHE75H3MFVTX4NYEPANCNFSM6AAAAAA44CIBSY> . You are receiving this because you commented.Message ID: ***@***.***>

[glesperance]

Chiming in that we had to backport our application to pydantic v1.10 to work with ray data.

[EKami]

Same issue here. I'm forced to stay on Pydantic 1.x because of Ray and it's hard to work with with FastAPI

[shavtge]

Also encountering compatibility challenges that necessitate remaining on Pydantic v1 to ensure functionality with Ray Serve. Means i have to re-write a lot of Pydantic v2 code

[jens-create]

Also encountering compatibility challenges but when using vllm that uses ray to serve model.

[edoakes]

Hi folks, I am currently working with the Pydantic team to make this happen. The underlying issue blocking us is here: pydantic/pydantic#6763.

I've merged fixes for this issue into pydantic & pydantic_core, so we are now waiting for their next release in order to unpin the dependency on our side.

[Archie181]

With the recent release of pydantic 2.5 will you be able to unpin the dependency now? Thanks

[shrekris-anyscale]

Yes, we're planning to unpin the dependency for Ray 2.9 (the next Ray release).

[Archie181]

Ok great, thanks, do you have a rough timeline on when that might be released?

[shrekris-anyscale]

It'll likely be out mid-December.

[Archie181]

Ok, thank you. Might it be unpinned sooner in a nightly?

[shrekris-anyscale]

Yes, it'll probably be unpinned in a nightly by the end of the month.

[ddelange]

ref #40451

[shrekris-anyscale]

We've merged the changes to make Ray compatible with Pydantic 2.5+. You can start using Pydantic 2.5+ with Ray 2.9, which should be out at the end of December.

These changes should also be in the Ray nightly, so feel free to try them out!