[RayJob][Status][13/n] Make suspend operation atomic by introducing the new status Suspending

[kevin85421]

Why are these changes needed?

The suspend operation should be atomic. In other words, if users set the suspend flag to true and then immediately set it back to false, either all of the RayJob's associated resources should be cleaned up, or no resources should be cleaned up at all. To keep the atomicity, if a RayJob is in the Suspending status, we should delete all of its associated resources and then transition the status to Suspended no matter the value of the suspend flag.

There is a breaking change in this PR:

• This PR doesn't send a StopJob request to stop the Ray job to simplify logic. Currently, Ray doesn't have a best practice to stop a Ray job gracefully. At this moment, KubeRay doesn't stop the Ray job before suspending the RayJob. If users want to stop the Ray job by SIGTERM, users need to set the Pod's preStop hook by themselves. I think this change is acceptable because the suspend operation has not been working for several releases already.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# ray-job.long-running.yaml: https://gist.github.com/kevin85421/68fadd3ec00fc75d112512954f62cacf
kubectl apply -f ray-job.long-running.yaml

# Wait for the JobDeploymentStatus to be `Running`.
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
# [Expectation]: Running

# Add a finalizer to the RayCluster
kubectl patch rayclusters.ray.io rayjob-sample-raycluster-kp7c4 --type='json' -p='[{"op": "add", "path": "/metadata/finalizers", "value":["ray.io/test"]}]'

# Set `suspend` to true in `ray-job.long-running.yaml` and then apply again.
kubectl apply -f ray-job.long-running.yaml
# [Expectation]: The K8s Job should be removed, but the RayCluster will still be there.

# The status should be `Suspending`.
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
# [Expectation]: Suspending

# Remove the finalizer
kubectl patch rayclusters.ray.io rayjob-sample-raycluster-kp7c4 --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

# The status should be `Suspended`
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
# [Expectation]: Suspended

# Set `suspend` to false in `ray-job.long-running.yaml` and then apply again.
kubectl apply -f ray-job.long-running.yaml
# [Expectation]: The K8s Job and the RayCluster should be created again.

# The status should finally be `Running` again.
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
# [Expectation]: Running

[kevin85421]

Without this PR, this function returns nil when the DeletionTimeStamp is not zero.

[gvspraveen]

do we document this anywhere other than here in code?

[kevin85421]

Currently not. I will streamline the end-to-end UX of RayJob with some Kubernetes maintainers from Google this quarter. I will document important information when we figure out the best practice.