v1.7.1

Bug fixes

• Fixed a bug where the kubectl plugin did not re-check RayJob status after losing the port-forward connection, which could leave kubectl ray job submit hanging. #5147
• Fixed a bug where the History Server could lose rotated Ray logs by collecting them only after they were overwritten. #5101
• Reduced History Server memory usage by caching the decoded snapshot struct in the session LRU instead of raw JSON bytes. #5208
• Fixed broken sample manifests. #5154
• Repinned the ML samples to a base Ray image with runtime-installed deps, since the rayproject/ray-ml:2.46.0.0e19ea-* tags were removed from Docker Hub and the samples failed with ImagePullBackOff. #5194

Breaking Changes

• Disabled History Server live-cluster support by default to avoid unintended proxying to running clusters. #5199

Changelog

• b82f8ca [release-1.7] cherry picks for v1.7.1 (#5175)
• ce058c1 chore: bump to 1.7.1 (#5318)

Full Changelog: v1.7.0...v1.7.1

v1.7.0

Highlights

Ray History Server (beta)

The Ray History Server is graduating to beta. Introduced as alpha in KubeRay v1.6, the History Server lets you view the Ray Dashboard and debug ephemeral clusters (such as those managed by RayJob) even after they have been terminated.

Since v1.6, the History Server has received major improvements:

• Performance and scalability: lazy loading, an LRU byte-based session cache, a disk-first event storage pipeline, and event compression significantly reduce memory usage for both the collector and the server.
• Ray token authentication support for both the collector and live-cluster proxying.
• Better UI and API coverage: a cluster selection page, task logs, and Serve / placement group / Ray Data endpoints polled by default.
• History Server images are now published to quay.io/kuberay (historyserver:v1.7.0 and collector:v1.7.0)

Try it here: History Server Quick Start Guide.

Automatic History Server Sidecar Configuration (alpha)

The operator can now automatically inject the History Server event collector sidecar into your RayCluster’s pods, no manual sidecar configuration needed. Enable the RayClusterHistoryServer feature gate on the operator and set spec.historyServerOptions on your RayCluster:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-historyserver
spec:
  historyServerOptions:
    collectorOptions:
      image: quay.io/kuberay/collector:v1.7.0

See ray-cluster.historyserver.yaml for a full example.

⚠️ Warning: RayClusterHistoryServer is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterHistoryServer=true  

RayService Incremental Upgrade graduates to Beta

The RayServiceIncrementalUpgrade feature gate is now beta and enabled by default. This feature allows RayService to perform zero-downtime upgrades by gradually shifting traffic to the new cluster using the Kubernetes Gateway API, instead of requiring 2x resources for a blue/green switch.

Since its Alpha release, the feature has received several improvements for more reliable and robust upgrades, including:

• Rollback support: Users can now revert the RayService spec mid-upgrade, with safe traffic shifting between the pending and active clusters.
• Improved reliability: Improved capacity management, traffic migration, failure handling, and cluster cleanup during incremental upgrades.
• E2E test coverage: Expanded test scenarios to improve coverage.

mTLS support via cert-manager (alpha)

KubeRay v1.7 can automatically provision and rotate certificates for mutual TLS between RayCluster’s pods using cert-manager.

⚠️ Warning: RayClusterMTLS is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterMTLS=true  

Enable mTLS feature with following to try this feature:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-mtls
spec:
  rayVersion: '2.55.1'
  tlsOptions:
    enabled: true

See ray-cluster.mtls.yaml for a full example.

NetworkPolicy support for RayCluster (alpha)

KubeRay v1.7 adds a new spec.networkPolicy field to RayCluster for network isolation. When set, the operator creates separate NetworkPolicies for head and worker pods, with configurable modes (DenyAll, DenyAllIngress, DenyAllEgress) and per-worker-group ingress/egress rules. Intra-cluster pod-to-pod traffic is always permitted.

Try it here: Network Policy Quick Start Guide.

⚠️ Warning: RayClusterNetworkPolicy is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayClusterNetworkPolicy=true  

Below is an example of enabling network policy feature:

apiVersion: ray.io/v1
kind: RayCluster
spec:
  networkPolicy:
    mode: DenyAll
   head:
      ingressRules:
      - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <prometheus-namespace>
        ports:
        - port: 8080
          protocol: TCP

See ray-cluster.network-policy-deny-all.yaml for an example.

GCS fault tolerance with embedded storage (alpha)

GCS fault tolerance no longer requires an external Redis. With the GCSFaultToleranceEmbeddedStorage feature gate enabled, you can persist GCS metadata in an embedded RocksDB store backed by a persistent volume on the head Pod:

spec:
  gcsFaultToleranceOptions:
    backend: rocksdb
    storage:
      size: 1Gi

⚠️ Warning: GCSFaultToleranceEmbeddedStorage is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=GCSFaultToleranceEmbeddedStorage=true  

See ray-cluster.embedded-gcs-ft.yaml for a full example.

Other Notable Features

• RayCluster now supports built-in Ingress configuration via headGroupSpec.ingressOptions with host, path, pathType, and tls fields.
• RayCronJob now supports the spec.timeZone field, and child RayJobs use deterministic names to prevent duplicate firings.
• The WorkerGroup API includes a new priority field for scheduler integrations.
• A new alpha feature gate SidecarSubmitterRestart prevents premature job termination during transient head Pod resource spikes in RayJob sidecar mode.
• The operator configuration supports defaultPodAnnotations and defaultPodLabels applied to all Ray Pods.
• Memory usage improvements in the operator: Pod informer cache selectors and more efficient pod resource calculation.
• kubectl ray logs support --gke-link to print a Cloud Logging link on GKE.
• Upgraded to Go 1.26; Grafana dashboards updated for Ray 2.56.

Breaking Changes

• Helm: in the kuberay-operator chart, metrics.serviceMonitor.selector has been renamed to metrics.serviceMonitor.additionalLabels, and empty labels are no longer emitted (#4979). Update your values file if you set this field.
• History Server (alpha to beta): configuration has changed since the v1.6 alpha. S3 credentials now use the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, ...) instead of AWS_S3ID/AWS_S3SECRET/AWS_S3TOKEN (#4665), the runtime-class flag has been replaced by --storage-backend (#5085), and the on-storage event layout has changed (#4670). Redeploy the collector and history server together on v1.7 and see the updated setup guide.

Deprecations

• The ray.io/v1alpha1 API version is now marked deprecated (#5122). Please migrate to ray.io/v1.

Changelog

• Disable RayMultiHostIndexing feature for TestReconcile_Multihost_Replicas by @Future-Outlier in #4583
• Disable the field alignment lint check by @jinbum-kim in #4560
• Mark RC releases as pre-release in GoReleaser by @Future-Outlier in #4590
• [RayCluster] Add example YAML for manually enabling Ray k8s auth by @andrewsykim in #4582
• [RayService] Rollback Support for Incremental Upgrades by @ryanaoleary in #4109
• [RayService] Promote Incremental Upgrade Feature to Beta by @ryanaoleary in #4599
• Revert "[RayService] Promote Incremental Upgrade Feature to Beta" by @Future-Outlier in #4602
• [Github Action] Skip krew-index update for pre-release tags by @Future-Outlier in #4587
• [Helm] update ray-cluster chart to apiVersion: v2 by @mboersma in #4593
• Update kind configs and docs to v1.29.0 by @mboersma in #4595
• [Helm] remove ingress template support for k8s < 1.19 by @mboersma in #4591
• [Helm] Update ray-cluster default resource values by @Future-Outlier in #4588
• [Chore] Use RayCluster name as ServiceAccount name for RBAC authentication by @JiangJiaWei1103 in #4611
• [History Server] Add guide to push image to Google Artifact Registry for History Server images by @chiayi in #4618
• [History Server] Fix API response format to match Ray Dashboard frontend schema by @Future-Outlier in #4615
• [Test][Release] Change upgrade test version to test upgrade from 1.5.1 to 1.6.0 by @Future-Outlier in #4623
• [History Server] Typo and minor fix for image guide readme by @chiayi in https://github.com/ray-proj...

v1.7.0-rc.0

Changelog

• d59cbd8 Fix rayClusterScaleExpectation deletion to use request object when instance is nil (#4039)
• 480e128 Inject the --block option to ray start command automatically (#932)
• 0a9a3e3 Make replicas configurable for kuberay-operator #4180 (#4195)
• 7850773 Remove ray-cluster.without-block.yaml (#675)
• 162cafc [PodPool-VK] add podpool vk README (#4250) (#4251)
• 38ac168 [Telemetry] Inject env identifying KubeRay. #562
• 97425fa AGC gateway api example (#4076)
• dc203e2 Add DeepSeek example RayService (#3838)
• 7c0aa63 Add FAQ page (#1150)
• 8ae788b Add Fake TPU e2e Autoscaling Test Cases (#2279)
• fea763a Add Google Artifact Registry image build/push guide (#4618)
• 506bea3 Add Google Cloud storage history server deployment example (#4641)
• dcb97ce Add Grafana Dashboard for KubeRay Operator (#3676)
• bf7e497 Add Helm chart unit tests to ray-cluster (#3374)
• 113909f Add Helm chart unittests to CI (#3280)
• cb505dd Add Helm values for ResourceClaims to RayCluster (#4290)
• d0e8b57 Add KubeRay e2e Test for custom idleTimeoutSeconds with v2 Autoscaler (#2725)
• 6818a08 Add KubeRay related blogs (#1147)
• 042e6b4 Add NumOfHosts to RayCluster helm-chart template (#1969)
• 5adc91a Add NumOfHosts to WorkerGroupSpec (CRD change only) (#1834)
• 6f4e680 Add Pod informer cache selector to reduce memory usage (#4761)
• 80a6d58 Add Ray cluster spec for TPU pods (#1292)
• abb5291 Add RayCluster YAML for verl example (#3833)
• 3a3e634 Add RayCluster manifest example for history server (#4642)
• f232b5b Add RayClusterProvisioned Condition Type (#2301)
• cbaf5d7 Add RayClusterReady Condition Type (#2271)
• 745e176 Add RayJob sample for JAX TPU profiling (#5059)
• 732453e Add RayJob training example using pytorch resnet image classifier (#2107)
• e6ff35b Add RayService IncrementalUpgrade E2E tests to Buildkite (#4497)
• a0afea2 Add RayService Manifests for Stable Diffusion TPU Examples (#2198)
• f384932 Add RayService finalizer (#4807)
• 044105c Add RayService incremental upgrade sample for guide (#4164)
• 753dc05 Add RayService sample test (#1377)
• c835117 Add TPU to Known Custom Accelerators for generated rayStartCommand (#2495)
• fe1ad90 Add NewClusterWithIncrementalUpgrade RayService rollback load tests (#4915)
• f05fa2e Add ray.io/originated-from labels (#1830)
• 87407ac Add a document for profiling (#1299)
• 08792ca Add a document to outline the default settings for rayStartParams in Kuberay (#1057)
• b92d95a Add a flag to enable/disable worker init container injection (#1069)
• 8851088 Add a grouping for 'google.golang.org/*' to avoid inconsistency between sub-projects (#3470)
• ceb9f01 Add a sample RayJob to fine-tune a PyTorch lightning text classifier (#1891)
• a851490 Add a test util function for killing the head Pod and wait (#3890)
• 073de1f Add a util function to convert string and bytes array (#2621)
• 3e20a9d Add a variant of the ray data processing job with GCSFuse CSI driver (#2401)
• 7fe4050 Add a warning to discourage users from launching a KubeRay-incompatible autoscaler. (#1102)
• efa2c1c Add additional KubeRay autoscaler e2e tests (#2761)
• 8b61b73 Add all and worker node type to kubectl ray log (#2442)
• 966d9b3 Add apply configurations to generated client (#1818)
• ef0129e Add basic Helm chart unittests for kuberay-operator (#3253)
• cd239ab Add basic e2e test for kubectl plugin (#2287)
• e1edb4c Add batch-scheduler option, deprecate enable-batch-scheduler option (#2300)
• e330c03 Add common containerEnv section to Helm Chart (#1932)
• 70194b7 Add compression functions (#4828)
• 70ef243 Add consistency check for deepcopy generated files (#1127)
• 36267ed Add dashboard component to master (#3566)
• cb2914d Add deletecollection for multi-namespace role (#2) (#2231)
• 8bb3222 Add dependabot.yml for enabling "Dependabot version updates" (#3357)
• 9a0b9d0 Add dnsConfig to head, worker and additional workers (#2377)
• 80ce664 Add documentation for API Server monitoring (#1479)
• 0bd28e7 Add documentations for the release process of Helm charts (#723)
• 7f3fe8b Add e2e KubeRay operator upgrade test (#3060)
• e9f3155 Add e2e test for kubectl ray job submit (#2614)
• 4fc48ce Add e2e test make sure resource quota error is surfaced (#3087)
• ff45923 Add end to end tests to apiserver (#1460)
• 4b0f7cb Add env and patch permission. (#740)
• 1bcfa9e Add env variable comment to kuberay-operator
• 54a9488 Add example YAML for manually enabling Ray k8s auth (#4582)
• d93c3c9 Add example and tutorial to explain how to create custom metrics for Prometheus (#914)
• 9198189 Add example in GKE to enable Ray resource isolation using cgroupsv2 and writable cgroup containers (#4236)
• a34a42a Add flag leader-election-namespace (#1624)
• a2ebc61 Add gofumpt instructions from internal doc (#1180)
• 1a544ab Add history server api for injecting collector sidecar (#4982)
• 044008d Add instruction to skip unit tests in DEVELOPMENT.md (#1171)
• abafd17 Add kubectl plugin with basic command and deprecate cli (#2243)
• 3e68606 Add kubectl ray cluster log command (#2296)
• 12babc8 Add kubectl ray create cluster (#2607)
• 61a282f Add kubectl ray delete rayservice/job/cluster (#2635)
• 8c64e60 Add kubectl-plugin pre-commit (#2255)
• 11c75ea Add kuberay operator servicemonitor (#3717)
• 25d5568 Add kubernetes dependency in python client library (#998)
• c8f826b Add kubernetes event to inform user of upgrade strategy (#2592)
• ea46c77 Add managedBy field to RayService (#4491)
• 106f8fd Add missing labels on RayCluster TPU manifests (#1987)
• 4a12d78 Add more grouping to resolve inconsistencies when bumping versions (#3554)
• 20ed3b3 Add no-session defaulting route and test for enter_cluster (#4907)
• 81e45da Add quay release for history server (#4980)
• 7856027 Add rayVersion in the RayCluster chart (#975)
• 4021766 Add rayjob yaml generation to ray job submit command (#2644)
• d22d752 Add release command and guidance for KubeRay cli (#834)
• e9544fc Add reminders to avoid RBAC synchronization bug (#576)
• 08da595 Add seccompProfile to KubeRay operator deployment for PSS compliance (#3931)
• 522807d Add seccompProfile.type=RuntimeDefault to kuberay-operator. (#1955)
• b5b4232 Add structured config and default sidecar container configuration (#1822)
• 58c2aad Add support for Ray token auth (#4179)
• 224a444 Add support for openshift routes (#1183)
• 43ed246 Add support for parsing neuron core resource limit and pass it as ray… (#2409)
• 2de3fe5 Add support for pvcs to apiserver (#1118)
• 3cc6116 Add support for tolerations, env, annotations and labels (#1070)
• aeba37e Add test for autoscaler and its desired state (#2601)
• 76633c5 Add test for configurable k8s job backoff...

v1.6.2

Bug Fixes

• Fix a bug where Redis clean up job would never execute when using RayService with Kueue (#4807)
• Add a timeout to enforce removal of Ray cluster finalizers. This prevents scenarios where Ray cluster deletion is blocked due to Redis clean up job not running. (#4836)

Changelog

• 7e31d3b [release-1.6] cherry picks for v1.6.2 (#4909)
• 598eb66 bump version to v1.6.2 (#4919)

v1.6.1

Bug fixes

• Fixed a bug where using spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavior in some cases with Kueue when queueing or pre-empting RayService objects #4739
• Fixed a bug where Volcano PodGroups can be deleted unconditionally. #4669

Changelog

• c9ddfc2 [Release][Helm] update KubeRay version to v1.6.1 (#4759)
• c2eea31 [release-1.6] cherry-pick RayService e2e test fixes (#4757)
• 947807e [release-1.6] cherry-pick bug fixes for v1.6.1 (#4744)

v1.5.2

Bug fixes

• Fixed a bug where using spec.rayClusterSpec.suspend in RayService did not trigger suspend on the underlying RayCluster. This can break suspend behavior in some cases with Kueue when queueing or pre-empting RayService objects #4739

Changelog

• 9f65eca [Release][Helm] update KubeRay version to v1.5.2 (#4758)
• 59caac7 [release-1.5] cherry-pick bug for v1.5.2 (#4751)
• 7e15b52 fix auto release CI (#4372)

v1.6.0

Highlights

Ray History Server (alpha)

KubeRay v1.6 introduces alpha support for the Ray History Server. This project enables users to collect and aggregate events from a Ray cluster, replaying them to restore historical snapshots of the cluster's state. By providing an alternative backend for the Ray Dashboard, the History Server allows users to view the Ray dashboard and debug ephemeral clusters (such as those managed via RayJob) even after they have been terminated.

Try the history server here: History Server Quick Start Guide.

⚠️ Warning: This feature is in alpha status, meaning future KubeRay releases may include breaking updates. We’d love to hear your experience with it! Please drop your feedback in this tracking issue to help us shape its development.

Ray Token Authentication using Kubernetes RBAC

Starting in KubeRay v1.6 and Ray v2.55, you can use Kubernetes RBAC to manage user access control to Ray clusters that have token authentication enabled. With this feature enabled, Ray will be configured to delegate token authentication to Kubernetes. This means you can use the same credentials used with Kubernetes to access Ray clusters and platform operators can use standard Kubernetes RBAC to control access to Ray clusters. See Configure Ray clusters to use Kubernetes RBAC authentication for a step-by-step guide.

You can now also reference Secrets containing static auth tokens for Ray cluster token authentication.

apiVersion: v1
kind: Secret
metadata:
  name: ray-cluster-token
type: Opaque
stringData:
  auth_token: "super-secret-example-token"
---
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: ray-cluster-with-auth
spec:
  authOptions:
    mode: token
    secretName: ray-cluster-token
  rayVersion: '2.53.0'
  headGroupSpec:
    rayStartParams: {}

RayCronJob

KubeRay v1.6 introduces the RayCronJob Custom Resource Definition (CRD), enabling users to schedule RayJobs on a recurring schedule using standard cron expressions. This is useful for periodic batch processing, scheduled training runs, or recurring data pipelines.

Try the Ray CronJob API here: Ray CronJob Quick Start Guide.

⚠️ Warning: RayCronJob is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:

--feature-gates=RayCronJob=true

Below is an example of the new custom resource:

apiVersion: ray.io/v1
kind: RayCronJob
metadata:
  name: raycronjob-sample
spec:
  schedule: "* * * * *"
  jobTemplate:
    entrypoint: python /home/ray/samples/sample_code.py
    shutdownAfterJobFinishes: true
    ttlSecondsAfterFinished: 600
    runtimeEnvYAML: |
      pip:
        - requests==2.26.0
        - pendulum==2.1.2
      env_vars:
        counter_name: "test_counter"
    rayClusterSpec:
      rayVersion: '2.52.0'
      headGroupSpec:
        ...
        ...

See ray-cronjob.sample.yaml for a full example.

RayJob Deletion Policy API

The RayJobDeletionPolicy feature gate is graduating to Beta and enabled by default. This feature enables a more advanced and flexible API for expressing deletion policies within the RayJob specification. This new design moves beyond the singular boolean field, spec.shutdownAfterJobFinishes, and allows users to define different cleanup strategies using configurable TTL values based on the Ray job's status.

Below is an example of how to use this new, flexible API structure:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: rayjob-deletion-rules
spec:
  deletionStrategy:
    deletionRules:
    - policy: DeleteWorkers
      condition:
        jobStatus: FAILED
        ttlSeconds: 100
    - policy: DeleteCluster
      condition:
        jobStatus: FAILED
        ttlSeconds: 600

See ray-job.deletion-rules.yaml for a comprehensive example.

Other Notable Features

• RayJob now supports spec.preRunningDeadlineSeconds to automatically mark failed jobs if they do not reach Running state within a specified timeout.
• RayService now supports spec.managedBy for improved support with Multi-Kueue
• The RayMultihostIndexing feature gate is graduating to Beta and enabled by default. This feature provides ordered replica and host index labels that are useful for managing Ray clusters for multi-host TPU/GPU workloads that require atomic scheduling and scaling. These labels are only applied if numOfHosts > 1 in worker group configuration.
• KubeRay v1.6 adds a new spec.upgradeStrategy field to RayCluster. Supported values are Recreate and None. The Recreate strategy is useful for automatically recreating all Ray cluster Pods when the Ray cluster spec changes. This upgrade strategy is not recommended if Ray cluster state needs to be persisted.
• RayService incremental upgrade feature (alpha status) now supports rollback.

Breaking Changes

When using Sidecar submission mode with RayJob, the Head Pod will no longer be automatically recreated after initial provisioning. This is because the submission container runs along the Head container and recreating the Pod would result in restarting the job entirely. See #4141 for more details.

CHANGELOG

• Add support for Ray token auth (#4179, @andrewsykim)
• feat: upgrade to Ray 2.52.0 to support token auth mode (#4152, @Future-Outlier)
• update minimum Ray version required for token authentication to 2.52.0 (#4201, @andrewsykim)
• add samples for RayCluster token auth (#4200, @andrewsykim)
• [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202, @Future-Outlier)
• [RayJob] Add token authentication support for All mode (#4210, @Future-Outlier)
• Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213, @Future-Outlier)
• [RayJob] Add token authentication support for light weight job submitter (#4215, @Future-Outlier)
• [RayCluster] make auth token secret name consistency (#4216, @fscnick)
• feat: kubectl ray get token command (#4218, @rueian)
• [RayService] auth token mode e2e test (#4225, @ryankert01)
• [e2e] RayJob Auth Mode E2E (#4229, @seanlaii)
• [e2e] Enhance RayCluster Auth E2E (#4231, @seanlaii)
• [RayJob] light weight job submitter upgrade to 1.5.1 to support auth token mode (#4235, @Future-Outlier)
• Support enabling RAY_ENABLE_K8S_TOKEN_AUTH (#4509, @andrewsykim)
• [Enhance] Refactor IsK8sAuthEnabled to accept AuthOptions and add token audience TODO (#4543, @Future-Outlier)
• [Enhancement] Add sample YAML for K8s token auth RayCluster (#4544, @Future-Outlier)
• improve API docs for EnableK8sTokenAuth (#4553, @andrewsykim)
• [ray-operator] add support for referencing Secret names for auth tokens (#4554, @andrewsykim)
• [Fix] Skip auth Secret reconciliation when K8s token auth is enabled (#4556, @Future-Outlier)
• [ray-operator][validation] K8s token auth mode does not support RayJob and RayService (#4562, @Future-Outlier)
• fix: upgrade Ray image in ray-cluster.auth.yaml to 2.53.0 to resolve dashboard 'Failed to load' error (#4310, @win5923)
• introduce historyserver directory and project structure (#4232, @andrewsykim)
• add the implementation of historyserver collector (#4241, @KunWuLuan)
• [history server] Remove go.work and go.work.sum to follow Go's best practices (#4301, @Future-Outlier)
• [history server] move storage interface (#4302, @my-vegetable-has-exploded)
• [historyserver][collector] Remove unused function processAllLogs (#4316, @ryankert01)
• [historyserver][collector] Add file-level idempotency check for prev-logs processing on container restart (#4321, @my-vegetable-has-exploded)
• [history server] Web Server + Event Processor (#4329, @Future-Outlier)
• [historyserver] Ensure at least one worker in sample RayCluster (#4330, @Future-Outlier)
• historyserver: remove unused function in RayLogHandler (#4336, @AndySung320)
• [history server][collector] Fix getJobID for job event collection (#4342, @Future-Outlier)
• [KubeRay Dashboard][Feature] Integrate History Server into KubeRay Dashboard (#4395, @CheyuWu)
• [Feat] [history server] Enable running the history server outside the K8s cluster (#4404, @JiangJiaWei1103)
• [history server][collector] Add WaitGroup for graceful shutdown (#4409, @fweilun)
• [1/N][Feature][history server] support endpoint /api/v0/logs/file (#4411, @machichima)
• [history server][storage] Add Azure Blob Storage support (#4413, @ikchifo)
• [historyserver][collector] use filepath functions to handle file path (#4417, @400Ping)
• [Feature][history server] support endpoint /api/cluster_status (#4421, @justinyeh1995)
• [1/N] [history server] Support job event processing and endpoint (#4422, @chiayi)
• [historyserver] implement grafana health for live session (#4425, @fscnick)
• [history server][collector] Enable run with race condition checker (#4430, @machichima)
• [Feat] [history server] Add node endpoint (#4436, @JiangJiaWei1103)
• [Feature][history server] support endpoint /api/v0/tasks/timeline (#4437, @AndySung320)
• [history server] Add logs/file resolution and logs/stream endpoint (#4456, ...

v1.6.0-rc.0

[Release] Update KubeRay version references for 1.6.0 (#4586)

* [Release] Update KubeRay version references for 1.6.0

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

---------

Signed-off-by: Future-Outlier <eric901201@gmail.com>

v1.5.1

v1.5.1

Highlights

This release adds support for Ray token authentication using RayCluster, RayJob and RayService.

You can enable Ray token authentication using the following API:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: ray-cluster-with-auth
spec:
  rayVersion: '2.52.0'
  authOptions:
    mode: token

You must specify spec.rayVersion to 2.52.0 or newer. See full example at ray-cluster.auth.yaml.

Bug fixes

• Fix a bug in the NewClusterWithIncrementalUpgrade strategy where the Active (old) cluster's Serve configuration cache was incorrectly updated to the new ServeConfigV2 during an upgrade. #4212 @ryanaoleary
• Surface pod-level container failures to RayCluster status #4196 @spencer-p
• Fix a bug where RayJob status is not updated if failure happens in Initializing phase #4191 @spencer-p
• Fix a bug where RayCluster status was not always propagated to RayJob status #4192 @machichima

v1.5.0

Highlights

Ray Label Selector API

Ray v2.49 introduced a label selector API. Correspondingly, KubeRay v1.5 now features a top-level API for defining Ray labels and resources. This new top-level API is the preferred method going forward, replacing the previous practice of setting labels and custom resources within rayStartParams.

The new API will be consumed by the Ray autoscaler, improving autoscaling decisions based on task and actor label selectors. Furthermore, labels configured through this API are mirrored directly into the Pods. This mirroring allows users to more seamlessly combine Ray label selectors with standard Kubernetes label selectors when managing and interacting with their Ray clusters.

You can use the new API in the following way:

apiVersion: ray.io/v1
kind: RayCluster
spec:
  ...
  headGroupSpec:
    rayStartParams: {}
    resources:
      Custom1: "1"
    labels:
      ray.io/zone: us-west-2a
      ray.io/region: us-west-2
  workerGroupSpec:
  - replicas: 1
    rayStartParams: {}
    resources:
      Custom1: "1"
    labels:
      ray.io/zone: us-west-2a
      ray.io/region: us-west-2

RayJob Sidecar submission mode

The RayJob resource now supports a new value for spec.submissionMode called SidecarMode.
Sidecar mode directly addresses a key limitation in both K8sJobMode and HttpMode: the network connectivity requirement from an external Pod or the KubeRay operator for job submission. With Sidecar mode, job submission is orchestrated by injecting a sidecar container into the Head Pod. This solution eliminates the need for an external client to handle the submission process and reduces job submission failure due to network failures.

To use this feature, set spec.submissionMode to SidecarMode in your RayJob:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: my-rayjob
spec:
  submissionMode: "SidecarMode"
  ...

Advanced deletion policies for RayJob

KubeRay now supports a more advanced and flexible API for expressing deletion policies within the RayJob specification. This new design moves beyond the singular boolean field, spec.shutdownAfterJobFinishes, and allows users to define different cleanup strategies using configurable TTL values based on the Ray job's status.

This API unlocks new use cases that require specific resource retention after a job completes or fails. For example, users can now implement policies that:

• Preserve only the Head Pod for a set duration after job failure to facilitate debugging.
• Retain the entire Ray Cluster for a longer TTL after a successful run for post-analysis or data retrieval.

By linking specific TTLs to Ray job statuses (e.g., success, failure) and strategies (e.g. DeleteWorkers, DeleteCluster, DeleteSelf), users gain fine-grained control over resource cleanup and cost management.

Below is an example of how to use this new, flexible API structure:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: rayjob-deletion-rules
spec:
  deletionStrategy:
    deletionRules:
    - policy: DeleteWorkers
      condition:
        jobStatus: FAILED
        ttlSeconds: 100
    - policy: DeleteCluster
      condition:
        jobStatus: FAILED
        ttlSeconds: 600
    - policy: DeleteCluster
      condition:
        jobStatus: SUCCEEDED
        ttlSeconds: 0

This feature is disabled by default and requires enabling the RayJobDeletionPolicy feature gate.

Incremental upgrade support for RayService

KubeRay v1.5 introduces the capability to enable zero-downtime incremental upgrades for RayServices. This new feature improves the upgrade process by leveraging the Gateway API and Ray autoscaling to incrementally migrate user traffic from the existing Ray cluster to the newly upgraded one.

This approach is more efficient and reliable compared to the former mechanism. The previous method required creating the upgraded Ray cluster at its full capacity and then shifting all traffic at once, which could lead to disruptions and unnecessary resource usage. By contrast, the incremental approach gradually scales up the new cluster and migrates traffic in smaller, controlled steps, resulting in improved stability and resource utilization during upgrade.

To enable this feature, set the following fields in RayService:

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: example-rayservice
spec:
  upgradeStrategy:
    type: "NewClusterWithIncrementalUpgrade"
    clusterUpgradeOptions:
      maxSurgePercent: 40 
      stepSizePercent: 5  
      intervalSeconds: 10
      gatewayClassName: "cluster-gateway"

This feature is disabled by default and requires enabling the RayServiceIncrementalUpgrade feature gate.

Improved multi-host support for RayCluster

Previous KubeRay versions supported multi-host worker groups via the numOfHosts API, but this capability lacked fundamental capabilities required for managing multi-host accelerators. Firstly, it lacked logical grouping of worker Pods belonging to the same multi-host unit (or slice). As a result, it was not possible to run operations like “replace all workers in this group”. In addition, there was no ordered indexing, which is often required for coordinating multi-host workers when using TPUs.

When using multi-host in KubeRay v1.5, KubeRay will automatically set the following labels for multi-host Ray workers:

labels:
  ray.io/worker-group-replica-name: tpu-group-af03de
  ray.io/worker-group-replica-index: 0
  ray.io/replica-host-index: 1

Below is a description of each label and its purpose:

• ray.io/worker-group-replica-name: this label provides a unique identifier for each replica (i.e. host group or slice) in a worker group. The label enables KubeRay to rediscover all other pods in the same group and apply group operators.
• ray.io/worker-group-replica-index: this label is an ordered replica index in the worker group. This label is particularly important for cases like multi-slice TPUs, where each slice must be aware of its slice index.
• ray.io/replica-host-index: this label is an ordered host index per replica (host group or slice).

These changes collectively enable reliable, production-level scaling and management of multi-host GPU workers or TPU slices.

This feature is disabled by default and requires enabling the RayMultiHostIndexing feature gate.

Breaking Changes

For RayCluster objects created by a RayJob, KubeRay will no longer attempt to recreate the Head Pod if it fails or is deleted after its initial successful provisioning. To retry failed jobs, use spec.backoffLimit which will result in KubeRay provisioning a new RayCluster.

CHANGELOG

• [release-1.5] update version to v1.5.0 (#4177, @andrewsykim)
• [CherryPick][Feature Enhancement] Set ordered replica index label to support mult… (#4171, @ryanaoleary)
• [releasey-1.5] update version to v1.5.0-rc.1 (#4170, @andrewsykim)
• [release-1.5] fix: dashboard build for kuberay 1.5.0 (#4169, @andrewsykim)
• [release-1.5] update versions to v1.5.0-rc.0 (#4155, @andrewsykim)
• [Bug] Sidecar mode shouldn't restart head pod when head pod is delete… (#4156, @rueian)
• Bump Kubernetes dependencies to v0.34.x (#4147, @mbobrovskyi)
• [Chore] Remove duplicate test-e2e-rayservice in Makefile (#4145, @seanlaii)
• [Scheduler] Replace AddMetadataToPod with AddMetadataToChildResource across all schedulers (#4123, @win5923)
• [Feature] Add initializing timeout for RayService (#4143, @seanlaii)
• [RayService] Support Incremental Zero-Downtime Upgrades (#3166, @ryanaoleary)
• Example RayCluster spec with Labels and label_selector API (#4136, @ryanaoleary)
• [RayCluster] Fix for multi-host indexing worker creation (#4139, @chiayi)
• Support uppercase default resource names for top-level Resources (#4137, @ryanaoleary)
• [Bug] [KubeRay Dashboard] Misclassifies RayCluster type (#4135, @CheyuWu)
• [RayCluster] Add multi-host indexing labels (#3998, @chiayi)
• [Grafana] Use Range option instead of instant for RayCluster Provisioned Duration panel (#4062, @win5923)
• [Feature] Separate controller namespace and CRD namespaces for KubeRay-Operator Dashboard (#4088, @400Ping)
• Update grafana dashboards to ray 2.49.2 + add README instructions on how to update (#4111, @alanwguo)
• fix: update broken and outdated links (#4129, @ErikJiang)
• [Feature] Provide multi-arch images for apiserver and security proxy (#4131, @seanlaii)
• test: add LastTransition to fix test (#4132, @machichima)
• Add top-level Labels and Resources Structed fields to HeadGroupSpec and WorkerGroupSpec ([#4106](#4...