[RayService][Health-Check][1/n] Offload the health check responsibilities to K8s and RayCluster

[kevin85421]

Why are these changes needed?

RayService features a health check mechanism to monitor the status of the dashboard agent on the Ray head and the status of Ray Serve applications. It will trigger a new RayCluster preparation when the RayService controller thinks that the RayCluster is not healthy.

Almost no users benefit from this mechanism, and many have complained about why KubeRay created a new RayCluster automatically. Most users do not have enough computing resources to run two RayClusters simultaneously. In addition, maintaining the RayService health check mechanism is quite challenging due to the numerous interconnected health checks in KubeRay, including: (1) the RayCluster controller, (2) K8s readiness/liveness probes, (3) Raylet, (4) Ray Autoscaler, and (5) the RayService controller. That's why I decide to offload the health check responsibilities to K8s and RayCluster controller.

This PR avoids calling the function markRestart based on data plane status.

Related issue number

• [Hotfix][Bug] Avoid unnecessary zero-downtime upgrade #1581
• [RayService] Revisit the conditions under which a RayService is considered unhealthy and the default threshold #1293
• [Bug] Long image pull time will trigger blue-green upgrade after the head is ready #1231
• https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1700143910045419?thread_ts=1699886904.977709&cid=C02GFQ82JPM

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# Step 1: Create a RayService with (1) serviceUnhealthySecondThreshold: 90 and (2) deploymentUnhealthySecondThreshold: 30.
# https://gist.github.com/kevin85421/deba2e8c8e18455d6911cfe90a03e493

# Step 2: Delete the dashboard agent process after the Ray Serve applications are ready
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl exec -it $HEAD_POD -- bash
kill $DASHBOARD_AGENT_PID

# Step 3: Wait for 3 mins, and no new RayCluster preparation should be triggered.

• Note that the dashboard agent status should be monitored by K8s Pod probes. However, KubeRay currently only injects probes to Pods when GCS FT is enabled. Hence, I will open a following PR to add probes to Ray Pods no matter whether GCS FT is enabled or not.

[architkulkarni]

Looks good to me!