Ray-2.57.0

Highlights

• Ray Data: In this Ray release, we've enabled DataSourceV2 by default (#64821), so read_parquet and friends use the new scan/listing infrastructure with row-group-aware chunking and predicate splitting. Hash Shuffle V2 eliminates the aggregator actor pool. V1 had to provision that pool up front from an estimate of the input size, and its actors accumulated partition shards in actor heap memory, invisible to Ray and unspillable, until finalization. V2 replaces it with two stateless task-based operators, ShuffleMapOp --> ShuffleReduceOp, that pass shards through the object store, so intermediate state spills under pressure and no capacity has to be reserved in advance. The map/reduce barrier itself remains in both designs.
• HashShuffleV2 supports join (#63598, #64538, #64687). This lets shuffles reuse standard map/reduce scheduling, backpressure, and resource accounting.
• Ray Serve: The HAProxy ingress is now distributed as the ray-haproxy PyPI package instead of being compiled into images, and it is the default HAProxy binary (#64141, #64163, #64164). We've also added gRPC support to the HAProxy direct-ingress path, including streaming, metrics, and custom request IDs (#63735, #64310, #64166, #64112). For Ray Serve LLM, we've added experimental KV-cache-aware request routing that tracks replica KV state through an event plane, tokenizes before routing, and routes on prefill/decode token load (#64084, #64085, #64097, #64224, #64327, #64400). KV cache-aware routing’s complete support will land in 2.58.
• Ray Core: We've added an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selectable with RAY_gcs_storage=rocksdb and RAY_gcs_storage_path (#63657). GCS fault tolerance no longer requires an external Redis instance. We've also added a public API for topology-aware scheduling (#63479, #63740).

Ray Data

🎉 New Features

• Enable DataSourceV2 by default via DataContext.use_datasource_v2 (#64821)
• New task-based hash shuffle v2 (ShuffleMapOp → ShuffleReduceOp) with join, multi-input reduce, downstream map fusion, and reducer remote args, behind an env flag (#63598, #64538, #64687, #64438, #64302, #64532, #64481)
• Add a Catalog abstraction with a UnityCatalog implementation that can be passed to read_*, and Unity Catalog write support for Parquet and Iceberg (#64193, #64519)
• Add read_zarr for Zarr datasets (#63003) and read_lerobot for LeRobot v3 datasets (#63821)
• Add PushdownCountFiles optimization to answer count() from Parquet footers (#64763)
• Add common subexpression elimination to the expression optimizer (#63974)
• Add GPU support for Aggregate (#63708)
• Make dataset iteration metrics queryable per split (#64608)
• Add custom operator stats to capture worker-side metrics during task execution (#64221)
• Refactor usage collection into an extensible UsageCallback (#64500)
• Export from_blocks from ray.data (#64127)
• Add support for rapidsmpf-26.4.0 (#64324)

💫 Enhancements

• Hide Ray Data internal frames from user-code error tracebacks (#64587)
• Run projection/predicate pushdown before limit pushdown, block optimization of non-deterministic expressions, and make fuse checks consistent between the logical and physical optimizers (#64651, #64165, #63936)
• Add a two-mode block metadata fetch behind a single MetadataFetcher interface (#64378)
• Use work stealing for file listing and a dynamic work queue for traversals (#64675, #64388)
• Speed up ExecutionResources and the reservation/budget loops on the scheduling hot path (#63964)
• Fix O(n^2) schema reconciliation in unify_schemas and avoid per-column Series materialization in tensor-column casting (#64555, #64038)
• Decouple collate and memory pinning in iter_torch_batches, add per-stage training-thread blocking attribution, bound in-flight iter_threaded items, and finalize after reordering under preserve_order (#64653, #64183, #64219, #64282)
• Add default logical memory for map operators, wire the materializing-op filter into OpResourceAllocator budgeting, and move estimate_object_store_usage into the physical op (#63814, #63665, #63961)
• Decrease the downstream backpressure ratio to 2.0 (#64352)
• Route native S3FileSystem downloads to the PyArrow threaded path (#64089)
• Size hash-shuffle aggregators via bounded online sampling and add a timeout to the batched ray.get when fetching partitions (#63929, #64256)
• Replace manual block reference accounting with Ray Core out-of-scope object callbacks (#64011, #64157, #64191)
• Default read_numpy to allow_pickle=False and make it manually configurable (#64684)
• Make write_lance(mode=CREATE) error instead of silently overwriting (#64364)
• Expand DistributionTracker with merge() and p25/p75, and add dead node counts and detected issues to usage collection (#64074, #64459, #64198)
• Remove cluster autoscaler v1, rename the subcluster label key to ray-subcluster, avoid scaling nodegroups dedicated to the head node, and quiet autoscaling coordinator logs (#64380, #64003, #63918, #63534)
• Delay the "cluster resources not enough" warning until an operator is persistently starved, and lower the high-memory warning threshold (#63969, #64124)
• Remove ExecutionPlan, _num_outputs, batch_format on AllToAllOperators, and InheritBatchFormatRule; use input_dependencies in logical operators (#63662, #64167, #64152, #64149, #64148)
• Deprecate low-level scheduling APIs (DataContext.scheduling_strategy, actor_locality_enabled, exclude_resources, local://) ahead of the actor-only rearchitecture (#64632)
• Migrate the Daft dependency from getdaft to daft (#64240)
• Migrate apply_chat_template/tokenize/detokenize callers to *_stage form, and remove PrepareImageStage while deprecating the image row column (#63590, #63570)

🔨 Fixes

• Fix TensorDtype.__from_arrow__ crash on empty tensor columns (#64767)
• Fix Arrow-backed to_pandas regressions with an opt-out flag and int/float block overflow handling (#64768)
• Fix read-only hash array crash in hash partition (#64552, #64559)
• Fix streaming-generator hang on a plasma-resident return object (#64386)
• Fix Parquet write overflow for string/binary columns larger than 2 GiB (#64212)
• Fix Parquet sampling hang on a permanent OSError (#64342)
• Fix the BigQuery datasink to handle empty blocks (#60797)
• Make PandasBlock.size_bytes deterministic (#64393)
• Propagate branch in _append_and_commit() for the Iceberg overwrite save mode (#63922)
• Propagate isolate_read_workers to DatasourceV2 (#65191, #65207)
• Fix cross-region S3 bucket access in the obstore download path (#63890)
• Restore safe_round in the ExecutionResources hot path (#64296)
• Log both the wrapped and wrapper exception in iterate_with_retry (#64639)
• Add tf-keras to the text_embedding pip packages (#64889, #64968)
• Fix build_processor failing with vLLM >= 0.19, correct its config type hint, and add request_timeout_s to ServeDeploymentProcessor to prevent indefinite hangs (#64337, #64098, #64496)
• Fix type hints in throughput_solver.py (#64289)

📖 Documentation

• Add a guide on avoiding out-of-memory errors and improve the existing OOM docs (#64046, #64297)
• Document the best practice for isolating Ray Data workers from training nodes (#64588)
• Document running two datasets in one cluster (#64200)
• Move the read_zarr guide back into its docstring (#64409)
• Remove Ray Data and DAG docstring ignores from pydoclint (#63634)

Ray Serve

🎉 New Features

• Distribute HAProxy through the ray-haproxy PyPI package, make it the default binary, and remove the from-source build (#64141, #64163, #64164)
• Add gRPC support to the HAProxy direct ingress, including streaming, unary-unary metrics, and custom request IDs (#63735, #64310, #64166, #64112)
• Emit request and system metrics from HAProxy (#64329, #64255)
• Support root_path in the HAProxy ingress (#64295)
• Add a per-replica metadata hook exposed on ReplicaSelection (#63948)

💫 Enhancements

• HAProxy stability: graceful drain waits for in-flight and queued requests, replicas quiesce before reporting shutdown complete, dispatches cancelled by replica gRPC stop are retried, reload takeover is verified by pid, and close-spread-time is set in the template (#63886, #63920, #63995, #64022, #64063, #63996)
• Make direct-ingress port reconcile incremental and skip it entirely when ingress membership is unchanged (#64508, #64676)
• Controller performance: O(1) version-filtered replica counts, in-place health-check reconcile in the deployment-state loop, and a fast-path orphaned-actor check (#64699, #64507, #64511)
• Freeze GC after replica initialization by default (#64742)
• Eager-load proxy imports (#64113)
• Pin the replica actor to bundle 0 of its placement group and track actor fallback_strategy on DeploymentSchedulingInfo (#62693, #64346)
• Deprecate HTTPOptions.location in favor of proxy_location, raise on non-zero HTTPOptions.num_cpus, and warn on other deprecated HTTP options (#64479, #64418, #63604)
• Raise a clear error when a custom request router or serve.multiplexed is used on an ingress deployment under direct ingress (#64211, #64045)
• Recover ingress request router pin-misses via the fallback proxy instead of returning 503 (#64218)
• Add RAY_SERVE_CONTROLLER_METRICS_INCLUDE_HIGH_CARDINALITY_TAGS to control controller metric tags (#63642)
• Default RAY_SERVE_PORT_QUARANTINE_S to hard-stop-after plus a margin (#64021)
• Defer the "application ready" log until proxies are ready (#64010)
• Dedup ingress replica and proxy metrics code, and migrate parse_uri from _private to _common (#64041, #64371)
• Enforce mypy and pyrefly type checking on clean Serve files (#64662)
• Fail loudly when subclassing @serve.ingress with a sync __init__ (#63413)
• Supply a safe default request id header value in direct ingress (#64118)

🔨 Fixes

• Fix serve.ingress(FastAPI()) pickling and include_router under FastAPI >= 0.137 (#64814, #64531)
• Fix an idle worker leak that caused OOM on the Ray head node (#63597)
• Fix serve.shutdown() skipping live shutdown when the cached controller client is stale after a driver reconnect (#64660)
• Preserve the deployment actor class across a config re-apply (#64411)
• Fix rank corruption on controller recovery after a lightweight reconfigure (#64636)
• Fix route=None crash in direct-ingress metrics reporting (#64645)
• Fix the direct-ingress backpressure counter leak on cancel-while-queued and clean up the leaked receive task on request timeout (#64348, #64123)
• Fix HAProxy healthz falling through to 404 when there are no backends (#64582)
• Guard request router backoff against OverflowError (#64539)
• Fix a slot leak race in SingletonThreadRouter.choose_replica (#63649)
• Fix a type regression in RAY_SERVE_HAPROXY_CLOSE_SPREAD_TIME_S (#64752)
• Fix Serve tests broken by protobuf 7 (#64362)

📖 Documentation

• Fix the RequestRouterConfig stats docstring to reference record_routing_stats (#64086)

Ray Train

🎉 New Features

• Add preemption handling: a preemption watcher for node-drain observability, with the signal fanned out to workers (#63807, #64099)

💫 Enhancements

• Make Train v1 and v2 use the autoscaling coordinator (#64824, #64936)
• Restrict ray.train.report(checkpoint) to in-band checkpoints only (#63645)
• Make Result.from_path read-only (#64340)
• Add compute-time metrics to tune_torch_benchmark.py (#64274)
• Update TensorFlow/Keras examples and tests for Keras 3 compatibility (#64276)

🔨 Fixes

• Fix CometLoggerCallback mutating the caller's result dict via result.pop() (#64570)
• Fix the WorkerMetricsCallback method name to match the WorkerCallback interface (#64568)

📖 Documentation

• Document the unsafe deserialization risk in TorchCheckpoint.get_model() and FrameworkCheckpoint.get_preprocessor() (#64586)
• Fix docstring style and enable --check-style-mismatch=True in pydoclint (#63988)
• Enumerate the accepted resource keys for ScalingConfig (#63449)

Ray Tune

💫 Enhancements

• Make BayesOptSearch float-hash precision configurable (#63914)
• Require optuna>=3.0.0 in OptunaSearch (#64242)

🔨 Fixes

• Respect max_concurrent_trials for custom searchers (#63770)

📖 Documentation

• Warn when BayesOptSearch stops early on convergence, and document patience/skip_duplicate (#64288)
• Restructure the Tune example pages and add PyTorch Lightning examples to the examples README (#63460, #63438)
• Document nested-metric keys in ResultGrid.get_best_result and add a conditional search-space example to tune.sample_from (#63445, #63443)

Ray LLM

🎉 New Features

• Add KV-cache-aware request routing: KVAwareRouter/KVRouterActor interfaces, replica tracking, an event plane, pre-routing tokenization, token-level request lifecycle tracking, and prefill/decode token-load-aware routing (#64084, #64085, #64097, #64224, #64327, #64400)
• Switch the Ray Serve SGLang engine to the sglang RayEngine, reach control-plane parity, and support Ray Serve direct streaming by serving SGLang's native OpenAI app (#62888, #63021, #64611)
• Add the MoRIIO KV-connector backend for prefill/decode disaggregation, with cross-node worker IP advertisement via a vLLM plugin (#63951, #64067)
• Enable the /classify and /pooling endpoints in direct-streaming mode (#64494)
• Add a Ray Serve LLM SGLang metrics dashboard (#64797, #65083)

💫 Enhancements

• Upgrade vLLM to 0.25.1 (#64082, #64483, #64697)
• Delegate prefill/decode orchestration to the KV-connector backend and avoid double prompt tokenization in disaggregation (#63950, #64049)
• Make request.request_id authoritative for the engine and stop clobbering an explicitly set request id with the Serve id (#63949, #64044)
• Parse the direct-streaming routing payload for body-aware routers, and add direct streaming session-affinity tests (#64328, #63927, #64266)
• Use asyncio.create_task instead of ensure_future in KVRouterActor (#64546)
• Add a cluster filter to the Serve LLM Grafana dashboard (#64596)
• Validate accelerator_type for CPU vLLM engine configs (#64235)
• Use model_id rather than the remote URI as the cache identifier in VLLMEngineConfig (#64110)
• Remove the Serve LLM config generator (#64075)

🔨 Fixes

• Fix malformed PromQL in the Serve LLM Grafana dashboard (#63893)

📖 Documentation

• Add a custom vLLM model guide and update the vLLM embedding model example (#64594, #64504)
• Improve the Serve LLM docs landing pages, add a config reference, and document direct streaming and the ingress request router (#64076, #63860)
• Fix 404s in the Serve LLM examples sidebar links (#64209)

Ray RLlib

🎉 New Features

• Add a pull-based EnvRunnerStateServer for async weight sync (#63849)
• Expose the count of EnvRunners dropped on timeout_seconds calls (#63493)

💫 Enhancements

• Upgrade the example vision stack to MobileNet v3 and remove MobileNet v2 torchhub usage (#64031, #64028)

🔨 Fixes

• Narrow from_checkpoint expectations to avoid silent failures (#63614)
• Require MultiRLModuleSpec.rl_module_specs to be a dict (#64785)
• Fix the perpetual pending state in offline evaluation during Tune (#64453)
• Fix RLlib with Tune by reverting the AlgorithmConfig.to_dict() change for the new API stack (#63695, #64501)
• Fix ONNX export under torch 2.9 by bumping onnxscript and migrating to the dynamo exporter (#64410, #64033)
• Apply observation filters in compute_single_action() on the old API stack (#64088)

📖 Documentation

• Revamp the RL modules docs and speed up the docs tests (#64505)
• Fix the outdated parametric-action-spaces link in rllib-algorithms (#63435)

Ray Core

🎉 New Features

• Add an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selected via RAY_gcs_storage=rocksdb and RAY_gcs_storage_path (#63657)
• Add a public API for topology-aware scheduling via topology_strategy (#63479, #63740)
• Implement backpressure for async streaming generators, add actor-level backpressure config, and add _num_objects_per_yield (#64383, #63310, #63943)
• Implement RayTaskEventRecorder as the first step of moving task events out of GCS (#64168)
• Drain the node on SIGTERM before shutting down ray start --block (#64454)
• Add tpu.dispatch syntax sugar and a JAX profiler for TPU (#64493, #62371)
• Publicize __ray_call__ as a DeveloperAPI for running closures on actors (#64367)
• Add the NVIDIA RTX PRO 6000 accelerator type and update the AMD accelerator manager (#64428, #62477)
• RDT: enable the LIBFABRIC backend for NIXL and upgrade NIXL to v1.2.0 (#62339, #63980)
• Add an owner-driven FreeLocalObjects RPC and remove the original FreeObjects RPC and ObjectEviction pubsub (#63218, #63181)

💫 Enhancements

• Actor scalability: prefill the placement-group bundle cache at creation to remove an actor submission bottleneck, and construct gRPC metric objects once per process instead of per request (#64649, #64732)
• Bound the GCS worker table by evicting old dead-worker entries, prioritized by exit type (#64612, #64729)
• Batch object-manager Pull RPCs by destination node and read spilled object chunks in bulk instead of byte-by-byte (#64225, #63830)
• Post all object pubsub publishes to the IO thread to relieve lock contention (#63983)
• Autoscaler: improve v2 scheduler performance for large clusters, handle deleted node types in the metrics reporter, and support env-var configuration for log rotation (#64175, #64184, #63955)
• Integrate IOContextMonitor into GCS, update its metrics, and exclude ray_syncer_io_context and task_io_context from health checks while raising the probe deadline to 30s (#63930, #63975, #64522, #64421)
• Continue the ClockInterface migration in the core worker and dependency-inject PeriodicalRunnerInterface (#63956, #63994, #64061, #64029)
• Enable process_group_cleanup_enabled by default and fix graceful-shutdown cleanup (#64407)
• Make owned object metrics lazy, compute per-component memory usage in MiB, scope prom_metrics_service_discovery.json to the session dir, and set the OpenTelemetry resource identity (#64093, #63932, #63850, #63921)
• Re-emit per-state task, actor, and placement group gauges every metrics tick and retain gauge observations for a TTL, so live series survive a missed scrape (#64524, #64633)
• Add the GPU UUID to the labels of GPU metrics (#65113, #65134)
• Log a hint to the driver when system overhead exceeds reserved system memory (#64270)
• Validate explicit zero resource isolation overrides and move memory monitors into common/monitor (#63864, #64491)
• Prioritize tasks over actors when killing under memory pressure (#64170)
• Bail on AdjustWorkerOomScore read failure and clamp the score to [-1000, 1000] (#62713)
• Cap the runtime_env parse_uri package name length to avoid ENAMETOOLONG (#64339)
• Scope the raylet's forced GC to backstop actor handle cycle deadlocks (#63939)
• Support joining fibers before exiting the fiber runner thread (#64405)
• Internal cleanups: remove callbacks from TaskToExecute, split task/resource preparation utils out of _raylet.pyx, consolidate the GCS dedicated io_context policy, and rename OnDemandBroadcasting (#64182, #63088, #63931, #63855)

🔨 Fixes

• Fix a Python 3.14 async-actor memory leak by re-anchoring the stack (#65177)
• Fix a deadlock between metric registration and collect() in OpenTelemetryMetricRecorder (#64946, #65094)
• Fix multi-threaded actor graceful shutdown (#64019)
• Streaming generator fixes: remove ref stream callbacks that caused a reconstruction hang, fail fast on a replay object-count mismatch, remove the implicit __dealloc__(), and honor timeout_s in the end-of-stream ray.get (#64581, #64394, #64333, #64014)
• Fix pipelining of actor argument fetches while an actor task is running (#63079)
• Fix a dangling reference in GetSchedulingClassDescriptor (#64707)
• Fix a pgid cleanup leak in the job supervisor (#64746)
• Fix the raylet drain path reading stale object store memory usage (#64415)
• Avoid a GCS crash on Redis connection loss in RedisResponseFn (#64204)
• Fix the actor working_dir being overridden by the job-level py_driver_sys_path (#63756)
• Fix RAY_CHECK failures caused by double ray.cancel() and keyboard interrupts (#63663)
• Fix event_logger and export_event_logger to flush all handlers safely (#63947)
• Default TaskProfileEvent.extra_data_ to {} to avoid a JSON decoding error in the State API, and normalize the cmdline field in the StatsPayload schema (#64589, #64286)
• Autoscaler v2: fix ALLOCATION_TIMEOUT worker replacement ordering and the RAY_STOP_REQUESTED → RAY_RUNNING fallback when drain has succeeded (#63815, #63424)
• Jobs: replace an assert with RuntimeError in put_status and avoid mutating runtime_env during submission (#64569, #63990)
• Improve the error message when RAY_ADDRESS is set to an HTTP address (#64180)
• Define WIN32_LEAN_AND_MEAN globally for Windows builds (#64361)
• Wait for subprocesses to terminate in ray.shutdown() (#63655)
• Fix the per-process GPU memory utilization dashboard stat (#61412)

📖 Documentation

• Update the OOM documentation to use resource isolation mode (#63652)
• Clarify the KubeRay autoscaler restartPolicy requirement (#64855)
• Document the name parameter for task.options() (#63450)
• Enforce Doxygen documentation style for coding agents and remove Ray Core docstring ignores from pydoclint (#64188, #63639)

Dashboard

🎉 New Features

• Implement the frontend UI for Platform Events (#63332)

💫 Enhancements

• Link workers to actor detail pages and handle nil IDs (#63795)
• Support a Grafana Cloud metrics fallback (#64138)
• Format RuntimeEnvState.creation_time_ms as a duration (#64207)
• Diagnose dashboard-agent event-loop stalls (#64436)
• Log errors when job submission fails in JobHead and stop configuring the root logger on module import (#64443, #64463)

🔨 Fixes

• Fix local file inclusion in the /logs API endpoint by rejecting absolute paths and .. components in LogAgentV1Grpc.ListLogs() (#64701)
• Fix TPU metrics (#63998)
• Fix task grouping by custom task name in the Dashboard API (#63820)
• Fix an "automaticaly" typo in the Grafana dashboard descriptions (#64614)

Ray Wheels and Images

• Build and publish Python 3.14 wheels and images, add py3.14 TPU base deps, refresh the py3.14 locks, and add a py3.14 smoke release test (#64754, #64750, #64857, #64938).
• Build and publish cu130 variants for the ray and ray-extra images (#63972, #63801).
• Add a py3.13 ray-ml image with torchft-nightly (#63587).
• Add ray-haproxy to the ray[serve] extra and base requirements, and bump it to 2.8.25 for CVE-2026-55203/55204 (#64141, #64430).
• Unify requirements_compiled across Python versions and upgrade the HuggingFace stack to datasets 4.x (#64257, #64054).
• Fix the vendored RocksDB build: use Make instead of Ninja to repair aarch64 core builds, drop -Wl,-pie from the TSAN link flags, and silence TSAN false-positive races (#64748, #64917, #64937, #64759).

Documentation

• Added AGENTS.md with the AI-assisted contribution policy, routed .claude/CLAUDE.md to it, and added a documentation style and grammar guide (#64419, #64518).
• Converted the ray-contribute developer guides from RST to MyST, soft-wrapped prose across doc/source, and added an rst-to-myst conversion skill (#64100, #64111, #64115, #64136, #64259, #64279, #64135).
• Added a unified APIs navigation section, generated API stubs without a full doc build, and greened the API-doc consistency check with reverse/dedup policies and a split whitelist (#64387, #64423, #64420, #64783, #64786, #64808, #64818).
• Added curated page descriptions for nav pages (SEO and llms.txt) and a custom 404 page with absolute URLs (#64330, #64603).
• Built Read the Docs incrementally from the doc build cache, repaired cache production, and skipped RtD PR builds for code-only PRs (#64277, #64414, #64482, #64599, #64547, #64480).
• Hardened doc template fetches: pinned to specific build ids, guarded zip extraction against path traversal, failed the build loudly on fetch failure, and removed outdated templates now synced at build time (#64637, #64669, #64629, #64635, #63971).
• Upgraded Sphinx to 8.2.3 on Python 3.11, decoupled Sphinx from the test/CI dependency stack, and fixed moved intersphinx inventory URLs (#64070, #63962, #64516).
• KubeRay: added an IPPR user guide, documented RayCronJob timezone support and autoscaler command overwrite, updated the RayJob agent sandbox docs, and refreshed the Serve High Throughput guide for Ray 2.56 (#63212, #64444, #63627, #64460, #64144).
• Added notebook tests for the lightweight Ray Core examples and recorded the Train example notebooks that are intentionally untested (#64313, #64318).
• Refreshed the ecosystem page, added a Ray + Agent Sandbox example, and renamed the AI chat widget to "Ray Docs" (#64156, #64047, #64351).