[Core] Failed to use uv

[hongbo-miao]

What happened + What you expected to happen

Got error when I switched from pip to uv:

Engine execution exited with unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1530, in run_flow
    ret_val = run_flow_sync(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1375, in run_flow_sync
    return engine.state if return_type == "state" else engine.result()
                                                       ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 350, in result
    raise self._raised
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 765, in run_context
    yield self
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 1373, in run_flow_sync
    engine.call_flow_fn()
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 785, in call_flow_fn
    result = call_with_parameters(self.flow.fn, self.parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/callables.py", line 208, in call_with_parameters
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/prefect/workflow/src/main.py", line 15, in hm_daft_analysis
    ray.init(
  File "/usr/local/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ray/_private/worker.py", line 1539, in init
    ctx = builder.connect()
          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ray/client_builder.py", line 175, in connect
    client_info_dict = ray.util.client_connect.connect(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ray/util/client_connect.py", line 55, in connect
    conn = ray.connect(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ray/util/client/__init__.py", line 233, in connect
    conn = self.get_context().connect(*args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ray/util/client/__init__.py", line 97, in connect
    self.client_worker._server_init(job_config, ray_init_kwargs)
  File "/usr/local/lib/python3.12/site-packages/ray/util/client/worker.py", line 860, in _server_init
    raise ConnectionAbortedError(
ConnectionAbortedError: Initialization failure from server:
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/util/client/server/proxier.py", line 707, in Datapath
    if not self.proxy_manager.start_specific_server(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/util/client/server/proxier.py", line 307, in start_specific_server
    serialized_runtime_env_context = self._create_runtime_env(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/util/client/server/proxier.py", line 264, in _create_runtime_env
    raise RuntimeError(
RuntimeError: Failed to create runtime_env for Ray client server, it is caused by:
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/agent/runtime_env_agent.py", line 391, in _create_runtime_env_with_retry
    runtime_env_context = await asyncio.wait_for(
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/agent/runtime_env_agent.py", line 357, in _setup_runtime_env
    await create_for_plugin_if_needed(
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/plugin.py", line 254, in create_for_plugin_if_needed
    size_bytes = await plugin.create(uri, runtime_env, context, logger=logger)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 316, in create
    uv_dir_bytes = await task
                   ^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 296, in _create_for_hash
    await UvProcessor(
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 204, in _run
    async with dependency_utils.check_ray(python, self._exec_cwd, logger):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/contextlib.py", line 217, in __aexit__
    await anext(self.gen)
  File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/runtime_env/dependency_utils.py", line 68, in check_ray
    raise RuntimeError(
RuntimeError: Changing the ray version is not allowed: 
  current version: 2.43.0, current path: /tmp/ray/session_2025-03-04_22-12-52_751827_1/runtime_resources/uv/92f24158ee01efc12a2b49f2cd65845239cc9738/virtualenv/lib/python3.12/site-packages/ray
  expect version: 2.43.0, expect path: /home/ray/anaconda3/lib/python3.12/site-packages/ray
Please ensure the dependencies in the runtime_env pip field do not install a different version of Ray.

Versions / Dependencies

• Ray cluster is using ray:2.43.0-py312-cpu
• Daft 0.4.7

Reproduction script

When use pip, it works well

import daft
import ray
from daft import DataCatalogType
from daft.io.catalog import DataCatalogTable


def hm_daft_analysis() -> None:
    ray_address = "ray://hm-ray-cluster-head-svc.production-hm-ray-cluster.svc:10001"
    ray.init(
        ray_address,
        runtime_env={
            "pip": [
                "getdaft[aws,deltalake,ray]==0.4.6",
            ],
        },
    )
    daft.context.set_runner_ray(ray_address)

    table = DataCatalogTable(
        catalog=DataCatalogType.GLUE,
        database_name="production_motor_db",
        table_name="motor_data",
    )

    df = daft.read_deltalake(table)
    df = df.where(df["_event_id"] == "ad7953cd-6d49-4929-8180-99555bebc255")
    df = df.collect()
    print(f"{df = }")


if __name__ == "__main__":
    hm_daft_analysis()

When I changed to

    ray.init(
        ray_address,
        runtime_env={
            "py_executable": "uv run --isolated",
            "uv": [
                "getdaft[aws,deltalake,ray]==0.4.6",
            ],
        },
    )

It throws error.

[pcmoritz]

Thanks for the report! The reason why you are running into this is because you are combining the uv run dependency management (from https://docs.ray.io/en/master/ray-core/handling-dependencies.html#using-uv-for-package-management), which runs the workers in a completely isolated environment with the "uv" runtime environment, which is a drop in replacement for the "pip" environment (using "uv pip" instead of "pip") -- it is kind of like the difference between "uv run" and "uv pip".

I would recommend to put the dependencies in a pyproject.toml and use the uv run method by itself -- the advantage it has over the "uv pip" method is there is complete isolation of the workers, whereas "uv pip" will use some mixture between system / conda and uv packages.

We can (and will) definitely give a better error message for this going forward :)