[RayJob] Transition to Complete if the JobStatus is STOPPED

[kevin85421]

Why are these changes needed?

The definition of terminal status for a Ray job is defined here. SUCCEEDED, FAILED, and STOPPED belong to terminal status which means that the Ray job can't transition to any other. Note that we don't stop the job for the suspend operation since #1798.

Without this PR, if you stop a running job, the RayJob will not enter Complete forever. Because the status will not transition to Complete, the TTL will not be triggered.

# Command to stop the Ray job
curl -X POST $RAY_DASHBOARD_URL/api/jobs/$YOUR_JOB_ID/stop

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# Create a long-running RayJob with this gist: https://gist.github.com/kevin85421/9cfa33c0c866dbf143e33651561e5719
# Important parts:
# (1) shutdownAfterJobFinishes: true
# (2) ttlSecondsAfterFinished: 10
# (3) 
# apiVersion: v1
# kind: ConfigMap
# metadata:
#   name: ray-job-code-sample
# data:
#   sample_code.py: |
#     import time
#     for i in range(10000):
#       print(i)
#       time.sleep(1)


# Port-forwarding
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl port-forward $HEAD_POD 8265 

# Get Job ID
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobId}'
# [Example output]: rayjob-sample-hls2j

# Stop the job
curl -X POST 127.0.0.1:8265/api/jobs/rayjob-sample-hls2j/stop

# The Kubernetes Job will be complete, and the RayJob will become `Complete`.
# Wait for $ttlSecondsAfterFinished seconds. The RayCluster will be cleaned up.

[architkulkarni]

Good catch! Ideally we would have a test for this behavior, we can leave it as a followup issue if you don't have bandwidth right now.

[kevin85421]

Ideally we would have a test for this behavior, we can leave it as a followup issue if you don't have bandwidth right now.

I will add a test in a follow-up PR. It may involve significant changes because it needs to execute a command in the head Pod, a process not currently used in the existing RayJob e2e tests.

[kevin85421]

Open an issue to track the progress #1866.