KubeRay v0.6.0 roadmap

[kevin85421]

Timeline

• v0.5.0: Apr 11, 2023
• Sprint 1 (6 weeks): Apr 11, 2023 ~ May 23, 2023
• Sprint 2 (6 weeks): May 23, 2023~ Jun 30, 2023 (expected)

Sprint 1 (Apr 11, 2023 ~ May 23, 2023)

Test infrastructure

• [Feature] Migrate CI infra from GitHub Actions to Buildkite #695

RayService Stability

• RayService + Autoscaler

  • [Bug] RayService restarts repeatedly with Autoscaler #1030
  • [Bug] RayService restarts repeatedly when a new config is applied #626
  • [RayService] Ray cluster keeps restarting when autoscaling with RayServices #649
  • [Bug] 401 errors for cluster autoscaler in ray service after apply update. #924

• RayService + GCS FT [P1]

  • [Bug][GCS FT] Worker pods crash unexpectedly when gcs_server on head pod is killed #1036

• Zero-downtime (when to trigger the new RayCluster preparation)

  • [Bug] RayService restarts repeatedly with Autoscaler #1030

• Test

  • [Feature] E2E Test plan for RayService #762 (autoscaler, fault tolerance, zero-downtime)

RayJob refactoring for stability

• Design doc to address the following issues

  • [Feature] RayJob support retry #669
  • [Bug] 409 conflicts cause two jobs creations #756

• Implementation: Sprint 2 or next quarter.

Sprint 2 (May 23, 2023~ Jun 30, 2023)

RayService refactoring for stability

• When to create a Kubernetes event?
• When to trigger a new RayCluster preparation?
  • [Bug] RayService restarts repeatedly with Autoscaler #1030
• When to update the custom resource status? [Move to sprint 1]
  • [Umbrella] [Feature] Refactor the status update for RayCluster / RayJob / RayService #983
  • RayService object's Status is being updated due to frequent reconciliation #1065
  • [RayService] Submit requests to the Dashboard after the head Pod is running and ready #1074
• What should be the same for active / pending RayClusters (e.g. head / serve service) ([Feature] rolling upgrade design and implementation for Kuberay #527 (comment))? What should be different? [P1]

Possible items for sprint 2 (depends on the progress)

• GCS FT: [Bug] Head pod is deleted rather than restarted when gcs_server on head pod is killed. #638
• New RayJob implementation
• Autoscaler: [WIP][Bug] Autoscaler issues #1048