[Bug][RayService] KubeRay does not recreate Serve applications if a head Pod without GCS FT recovers from a failure.

[kevin85421]

Why are these changes needed?

The function checkIfNeedSubmitServeDeployment is used to determine whether to update or create Ray Serve applications. Without this PR, this function returns true in the following two conditions:

(1) When there is no Serve config cache in the KubeRay operator, indicating that either a new RayCluster has just become ready or the KubeRay operator has crashed.

(2) When the Serve config in the cache differs from the RayService CR's ServeConfigV2, indicating an in-place Serve config update.

However, in the case of a head Pod crash and restart without GCS FT, the KubeRay operator will not send a request to create new Ray Serve applications because the Serve config cache remains the same as the RayService CR's ServeConfigV2.

Furthermore, the RayCluster will not be labeled as unhealthy because the function getAndCheckServeStatus does not handle the edge case when the number of Ray Serve applications is 0.

TODO

e2e test

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# Step 1: Install a KubeRay operator with an image built by this PR
# Step 2: Create a RayService (path: ray-operator/config/samples)
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Step 3: Forward the dashboard
kubectl port-forward --address 0.0.0.0 svc/rayservice-sample-head-svc 8265

# Step 4: Check http://127.0.0.1:8265/#/serve

# Step 5: Delete the head Pod
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl delete pod $HEAD_POD

# Step 6: Check the dashboard again after the head Pod restarts

• Step 4 screenshot
  
• Step 6 screenshot
  

Without this PR, you cannot see any Serve application in Step 6.

• Example:
  

[architkulkarni]

Looks good to me. For my understanding, what happens to the old Serve application in this case (when the head pod without GCS FT crashes and restarts)?

[kevin85421]

what happens to the old Serve application in this case (when the head pod without GCS FT crashes and restarts)?

Sync offline: When the head Pod without GCS FT crashes and restarts, the Ray head node will kill all workers because they are "unknown" workers for the head node. In addition, without this PR, the KubeRay operator will not create the Serve applications again.