[RayJob][Status][2/n] Redefine ready for RayCluster to avoid using HTTP requests to check dashboard status

[kevin85421]

Why are these changes needed?

#1674 introduces default readiness and liveness probes for Ray Pods, regardless of whether GCS fault tolerance is enabled. For the head Pod, the readiness probe verifies the status of GCS and Raylet. The GCS check utilizes the API endpoint from the Ray dashboard at DASHBOARD_ADDRESS:8265/api/gcs_healthz.

I checked with @architkulkarni and @shrekris-anyscale. I can imply that the Ray dashboard is ready for “Ray Job Submission” and Ray Serve RESTful API if DASHBOARD_ADDRESS:8265/api/gcs_healthz returns “success”. Hence, a "ready" RayCluster implies that all Ray Pods are running and ready, including Ray head.

• Check https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/580-pod-readiness-gates/README.md for the details of PodReady.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[architkulkarni]

Looks good to me!

Question: In the state "Wait for the RayCluster.Status.State to be ready", what is the JobDeploymentStatus?

[kevin85421]

Looks good to me!

Question: In the state "Wait for the RayCluster.Status.State to be ready", what is the JobDeploymentStatus?

It should be "Initializing".

• JobDeploymentStatus is set to Initializing after JobID and RayClusterName are set to avoid double creations.
• Then, KubeRay tries to create RayCluster and submit ray job only when RayCluster is "ready".

[kevin85421]

• Run the following command twice. Pass all tests (4 tests for each run).

  RAY_IMAGE=rayproject/ray:2.8.0 OPERATOR_IMAGE=controller:latest python3 tests/test_sample_rayjob_yamls.py 2>&1