[Feature] The default ImagePullPolicy should be IfNotPresent

[kevin85421]

Why are these changes needed?

Although we have autoscalerOptions.imagePullPolicy to overwrite the default imagePullPolicy (Always) in Autoscaler, there are still some reasons to update the default value for imagePullPolicy.

1. No Ray image with the tag latest is used in KubeRay. In that case, the default image pull policy for Pod should be IfNotPresent. See this doc for more details.

2. The default image pull policies for head Pod and worker Pods are IfNotPresent, but the default image pull policy is Always.

3. https://github.com/ray-project/ray/tree/master/python/ray/tests/kuberay => With this PR, we can run the tests locally without updating any line of code.

Related issue number

Closes #890

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

With this PR

# Step 1: Build your custom Ray image "ray:test"
# Step 2: Create a Kind cluster and load the Ray image into the Kind cluster
kind create cluster --image=kindest/node:v1.23.0
kind load docker-image ray:test

# Step 3: Build a KubeRay operator image for this PR "controller:latest"

# Step 4: Install KubeRay operator (with this PR)
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm install kuberay-operator kuberay/kuberay-operator --version 0.4.0 --set image.repository=controller,image.tag=latest

# Step 5: (path: ray-operator/config/samples) Create a Autoscaler RayCluster
# (1) set autoscalerOptions.image to "ray:test" (2) comment out autoscalerOptions.imagePullPolicy to "IfNotPresent"
kubectl apply -f ray-cluster.autoscaler.yaml

# Step 6: Check imagePullPolicy (The autoscaler container is the second container in the head Pod)
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].imagePullPolicy}{"\n"}{end}'

Without this PR

# Step 1: Build your custom Ray image "ray:test"
# Step 2: Create a Kind cluster and load the Ray image into the Kind cluster
kind create cluster --image=kindest/node:v1.23.0
kind load docker-image ray:test

# Step 3: Install KubeRay operator (without this PR)
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm install kuberay-operator kuberay/kuberay-operator --version 0.4.0

# Step 4: (path: ray-operator/config/samples) Create a Autoscaler RayCluster
# (1) set autoscalerOptions.image to "ray:test" (2) comment out autoscalerOptions.imagePullPolicy to "IfNotPresent"
kubectl apply -f ray-cluster.autoscaler.yaml

# Step 5: Check imagePullPolicy (The autoscaler container is the second container in the head Pod)
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].imagePullPolicy}{"\n"}{end}'

[kevin85421]

cc @DmitriGekhtman

Would you mind reviewing this PR? This is related to autoscaler. Thanks!