[RayService][Kueue] Support top-level Spec.Suspend

[kevin85421]

Why are these changes needed?

Kueue POC PR based on top-level suspend API: kubernetes-sigs/kueue#11264

• Fix Issue #4686 (rayservice): update RayCluster when Suspend toggles #4739 makes the RayService controller update the underlying RayCluster CRs when RayService.spec.RayClusterSpec.Suspend changes. However, this introduces multiple issues:
  • Issue 1: The RayService CR is no longer purely declarative.
    • For example, if we set suspend to true during upgrade, the pending RayCluster will be suspended.The active RayCluster will not be suspended because UpgradeInProgress is true (code). However, the active RayCluster will also not be deleted after the pending RayCluster is suspended, since the reconciliation will fail early because the head Pod of the pending RayCluster doesn't exist. That is, a "suspended" RayService still has a running active RayCluster.
    • Even if we prevent the reconciliation from returning early and suspend the active RayCluster, the behavior is still incorrect. For example, when Kueue sets suspend to false to admit the RayService again, both the pending and active RayCluster CRs will be created again. However, we don't need the pending RayCluster.
    • If the RayService is not undergoing an upgrade, the only RayCluster will be suspended, which is the expected behavior. That is, the goal state of a "suspended" RayService is different from that of a "suspended" RayService during an upgrade.
  • Issue 2: The CR status is incorrect.
  • Issue 3: Doesn't take the upgrade into consideration.
    • Ideally, the logic introduced by Fix Issue #4686 (rayservice): update RayCluster when Suspend toggles #4739 should keep the pending and active RayCluster CRs in sync on suspend either both false or both true (there are bugs that cause the two RayClusters' suspend fields to drift apart — see Issue 1 for details).
      • However, during an upgrade, the active RayCluster should have suspend: false, and the pending RayCluster should have suspend: true until Kueue admits it.
  • Issue 4: Kueue also injects scheduling information into the CR spec before admission. Therefore, suspending and then unsuspending the existing RayCluster CRs may trigger multiple unnecessary upgrades.
• Kueue also has its own issues: MultiKueue: Support Elastic RayService kubernetes-sigs/kueue#11102 (comment).

Due to the issues from KubeRay and Kueue mentioned above, the Kueue integration for RayService doesn't work, especially during upgrades.

Changes

This PR adds top-level RayService.Spec.Suspend and tightens the semantics of the existing nested Spec.RayClusterSpec.Suspend so the two stop fighting each other. Together they make Kueue-style suspension work cleanly for RayService.

1. Spec.RayClusterSpec.Suspend is now applied only at RayCluster creation

Background: changing RayService.Spec.RayClusterSpec.Suspend on a live RayService used to be propagated to the existing RayCluster on every reconcile, which fought Kueue any time it toggled Suspend directly on the cluster.

This PR changes the semantics:

• At creation time: Suspend is copied from the RayService onto the new RayCluster.
• After the RayCluster exists: hash comparisons (rayClusterSpecForHashing) ignore Suspend, so toggling it on the RayService never triggers a shouldUpdateCluster / shouldPrepareNewCluster path; and modifyRayCluster snapshots and re-applies the existing cluster's Suspend instead of letting the goal spec overwrite it. Ownership of Suspend on the RayCluster is delegated to Kueue (or whichever queueing controller owns it).

2. Top-level RayService.Spec.Suspend tears down all owned resources

When Spec.Suspend=true, the controller deletes everything it owns:

• All RayClusters labeled with the RayService association
• Head/serve Kubernetes Services
• Gateway/HTTPRoute (gated on RayServiceIncrementalUpgrade since those are only registered in the scheme when the feature gate is on)

The lifecycle is reported through two new conditions, Suspending and Suspended, and follows this state machine:

(no suspend) --Spec.Suspend=true--> Suspending --owned resources deleted--> Suspended --Spec.Suspend=false--> (no suspend)

Atomicity is enforced by treating the persisted Suspending condition as a commit point. The first reconcile that observes Spec.Suspend=true only stages the status transition — setting Suspending=True together with resetting ActiveServiceStatus, PendingServiceStatus, NumServeEndpoints, and ServiceStatus in the same Status().Update. The actual deletion runs on the next reconcile, once that commit is durable. If a deletion attempt errors out or Spec.Suspend is flipped back to false mid-way, Suspending stays True in storage and subsequent reconciles continue the cleanup.

handleSuspend only mutates Status; the caller persists changes via a single Status().Update.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • E2E tests (test/e2erayservice/rayservice_suspend_test.go):
    • TestRayServiceSuspendResume — happy path: Running → Suspended (all owned resources gone, status reset) → resume → Ready, serves traffic again.
    • TestRayServiceSuspendAtomic — mirrors the RayJob should be atomic pattern: pins the underlying RayCluster with a synthetic finalizer, then flips Spec.Suspend back and forth while asserting Consistently(Suspending=True); after removing the finalizer, verifies the original RayCluster is deleted and a different RayCluster eventually backs the Ready RayService.
    • TestRayServiceCreatedSuspended — Spec.Suspend=true from creation: no owned resources are ever created, Suspended=True is reached directly; flipping to false brings the service up normally.
    • TestRayServiceSuspendDuringUpgrade — triggers a zero-downtime upgrade, then suspends mid-upgrade: both active and pending RayClusters are deleted; resuming applies the upgraded spec.
  • Manual tests on a kind cluster (suspend/resume + suspend-during-upgrade).

[kevin85421]

Hi @mimowo @hiboyang,

This is a POC PR (kubernetes-sigs/kueue#11264) based on the top-level suspend API. Could you take a look at whether the API and contract proposed in this KubeRay PR make sense? You don't need to review the implementation details — reading the PR description and the e2e tests should be enough to understand the behavior.

If this makes sense to you, I will ping other KubeRay folks to review this PR. Thanks!

[mimowo]

Thank you @kevin85421 👍

cc @yaroslava-serdiuk who could likely provide the first review pass from the Kueue side.

[kevin85421]

Thanks @mimowo!

@yaroslava-serdiuk: The first step is to make sure the proposal in this KubeRay PR makes sense to you. Then I can ping other KubeRay folks to review it.

The Kueue PR kubernetes-sigs/kueue#11264 is a POC based on the API introduced in this KubeRay PR. It's not ready for review, but you can take a quick look to understand what the integration will look like.

Thanks!

[yaroslava-serdiuk]

Thanks @kevin85421 for the change! In general RayService.Spec.Suspend field makes sense to me.
Could you please clarify — what is the purpose of Spec.RayClusterSpec.Suspend field if there is RayService.Spec.Suspend field?

[hiboyang]

+1 for top-level RayService.Spec.Suspend, which will help Kueue MultiKueue to support RayService as well! Same question for Spec.RayClusterSpec.Suspend, do we still need it

[kevin85421]

Thanks @yaroslava-serdiuk @hiboyang for the review.

Could you please clarify — what is the purpose of Spec.RayClusterSpec.Suspend field if there is RayService.Spec.Suspend field?

Sure.

Context

Before we dive into more details, it will be helpful to understand the following context:

• RayService.Spec.RayClusterSpec:
  • The RayClusterSpec struct is defined in the RayCluster CRD. Therefore, RayService.Spec.RayClusterSpec.Suspend comes from the RayCluster CRD, not from RayService.
• RayService's zero-downtime upgrade
  • When users update RayService's spec (e.g., upgrade the image from rayproject/ray:2.X.0 to rayproject/ray:2.{X+1}.0), it will trigger a zero-downtime upgrade.
  • A new RayCluster (let's call it new_cluster below) will be created, and KubeRay will create Ray Serve applications in the new_cluster.
  • Then, KubeRay will delete the old_cluster.

RayService.Spec.Suspend / RayService.Spec.RayClusterSpec.Suspend

Behavior:

• RayService.Spec.Suspend:
  • true: delete all K8s resources owned by the RayService.
  • false: admit the RayService and consume the quota defined in RayService.Spec.RayClusterSpec.
• RayService.Spec.RayClusterSpec.Suspend
  • At creation time: RayService.Spec.RayClusterSpec.Suspend is copied from the RayService onto the new RayCluster.
  • After the RayCluster exists: RayService.Spec.RayClusterSpec.Suspend will be ignored, and Kueue's RayService controller manages the underlying RayCluster CRs' suspend values.

Example: kubernetes-sigs/kueue#11264:

1. A user creates a RayService CR. To simplify the explanation, we assume the RayService has only one head Pod and no worker Pods.
2. Kueue's webhook sets RayService.Spec.Suspend and RayService.Spec.RayClusterSpec.Suspend to true, suspending the RayService.
   1. RayService.Spec.Suspend: true, RayService.Spec.RayClusterSpec.Suspend: true
3. Kueue creates a Workload CR W1, which requires resources derived from RayService.Spec.RayClusterSpec (i.e., one head Pod).
4. W1 will be admitted if the ClusterQueue has enough resources to accommodate one head Pod. Admitting the RayService means setting RayService.Spec.Suspend to false.
   1. RayService.Spec.Suspend: false, RayService.Spec.RayClusterSpec.Suspend: true
5. Then, the KubeRay RayService controller will create a RayCluster CR (call it old_cluster in this example) with suspend=true, because RayService.Spec.RayClusterSpec.Suspend was set to true by Kueue's mutation webhook in Step 2.
   1. RayService.Spec.Suspend: false, RayService.Spec.RayClusterSpec.Suspend: true, old_cluster.Spec.Suspend: true
6. The Kueue RayService controller will set old_cluster.Spec.Suspend to false because W1 is admitted. See the function unsuspendAdmittedChildren in the POC PR for more details.
   1. RayService.Spec.Suspend: false, RayService.Spec.RayClusterSpec.Suspend: true, old_cluster.Spec.Suspend: false
7. The KubeRay RayCluster controller starts reconciling old_cluster.
8. A user updates the RayService CR's image field to trigger a zero-downtime upgrade.
9. The KubeRay RayService controller will create a new RayCluster CR (call it new_cluster in this example) with suspend=true, because RayService.Spec.RayClusterSpec.Suspend was set to true by Kueue's mutation webhook in Step 2.
   1. RayService.Spec.Suspend: false, RayService.Spec.RayClusterSpec.Suspend: true, old_cluster.Spec.Suspend: false, new_cluster.Spec.Suspend: true
10. The Kueue RayService controller will create a new Workload CR W2 (elastic workload slicing must be enabled). To be admitted, W2 requests the sum of the resources of old_cluster and new_cluster (i.e., two head Pods).
11. When the ClusterQueue has enough resources to admit W2, the Kueue RayService controller will set new_cluster.Spec.Suspend to false, and W1 will be marked as Finished.
    1. RayService.Spec.Suspend: false, RayService.Spec.RayClusterSpec.Suspend: true, old_cluster.Spec.Suspend: false, new_cluster.Spec.Suspend: false
12. The KubeRay RayCluster controller starts reconciling new_cluster.
13. When the new_cluster is ready to serve traffic, the KubeRay RayService controller will switch traffic to the new_cluster, then delete the old_cluster.
14. The Kueue RayService controller will reduce W2's resource requirement to one head Pod.