[release blocker][Feature] Only Autoscaler can make decisions to delete Pods

[kevin85421]

Why are these changes needed?

Without this PR, there are two mechanisms to delete Pods.

1. Autoscaler: When Autoscaler wants to scale down the cluster, it will send a patch to update replicas and workersToDelete fields in RayCluster CR. Then, KubeRay operator will delete all Pods in workersToDelete.
2. Random Pod deletion: If all Pods in workersToDelete have been deleted, and additional deletions are still required to reach the target state, the KubeRay operator will proceed to delete worker Pods randomly.

This PR provides a new behavior for Pod deletion:

• Case 1: If Autoscaler is disabled, we will always enable random Pod deletion.
• Case 2: If Autoscaler is enabled, the default behavior is disabling random Pod deletion. We also provide a feature flag ENABLE_RANDOM_POD_DELETE for users to go back to the old behavior if we ignore some edge cases.

Random Pod deletion is undesirable for numerous reasons. We should try to avoid it as much as possible. In addition, this behavior can avoid Case 4 mentioned in #1238 (comment).

Related issue number

#1238

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Test 1: No Autoscaler

helm install kuberay-operator kuberay/kuberay-operator --version 0.6.0-rc.0 --set image.repository=controller,image.tag=latest

# Create a RayCluster with 1 head and 1 worker
helm install raycluster kuberay/ray-cluster --version 0.6.0-rc.0

# Edit the worker's `replicas` to 2
# [Expected result]: 1 new worker will be created
kubectl edit rayclusters.ray.io raycluster-kuberay

# Edit the worker's `replicas` to 1
# [Expected result]: 1 random worker Pod will be deleted => random Pod deletion is enabled.
kubectl edit rayclusters.ray.io raycluster-kuberay

Test 2: Autoscaler

helm install kuberay-operator kuberay/kuberay-operator --version 0.6.0-rc.0 --set image.repository=controller,image.tag=latest

# Create an autoscaling-enabled RayCluster with 1 head and 1 worker
kubectl apply -f ray-cluster.autoscaler.yaml

# Execute a script in the head Pod.
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl exec -it $HEAD_POD -- bash

# https://gist.github.com/kevin85421/c35b69ff99cd30c55abbe4d5ccf2a18e
# Create 3 actors and each one requires a CPU, so Autoscaler scales up to 2 workers. (1 head = 1 CPU, 2 workers = 2 * 1 CPU)
python script.py 

# After `script.py` finishes, it will scale down to 1 worker because `minReplicas` is 1.

[wjzhou-ep]

Thank you very much about this new feature. I do think this is a really good idea!!!

One source of truth is always preferable.

[sihanwang41]

nit: rephrase the log statement.
"Randomly pick xx workers to delete from worker group xxx."

[sihanwang41]

nit: Ray Autoscaler. (if that is what you mean :) )