[Feature] Add a flag to make zero downtime upgrades optional

[kevin85421]

Why are these changes needed?

For LLM serving, some users might not have sufficient GPU resources to run two RayClusters simultaneously.
Therefore, KubeRay offers ENABLE_ZERO_DOWNTIME as a feature flag for zero-downtime upgrades.

Related issue number

Closes #1476

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# Modify `values.yaml` to disable zero-downtime upgrade
# - name: ENABLE_ZERO_DOWNTIME
#   value: "false"

# (Path: helm-chart/kuberay-operator)
helm install kuberay-operator . --set image.repository=controller,image.tag=latest

# Install a RayService.
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Forward the port for serving in a new terminal
kubectl port-forward svc/rayservice-sample-serve-svc 8000

# Send a request to the serve port.
curl -X POST -H 'Content-Type: application/json' localhost:8000/fruit/ -d '["PEAR", 12]'
# [Expected output]: 12

# Test in-place update
# Update the price of `PearStand` from 1 to 2.
#  - name: PearStand
#     num_replicas: 1
#     user_config:
#         price: 2
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Send a request to the serve port again.
curl -X POST -H 'Content-Type: application/json' localhost:8000/fruit/ -d '["PEAR", 12]'
# [Expected output]: 24

# Test zero-downtime upgrade.
# Set `rayVersion: '2.100.0'` in the RayService YAML to trigger the Zero-downtime upgrade.
kubectl apply -f ray_v1alpha1_rayservice.yaml

# [Expected result]: No new RayCluster will be created.

# Check logs
kubectl logs kuberay-operator-8fd6754b4-zvlxb | grep "Zero-downtime upgrade is disabled"
# You should see something like:
# 2023-10-25T18:08:47.993Z        INFO    controllers.RayService  Zero-downtime upgrade is disabled (ENABLE_ZERO_DOWNTIME: false). Skip preparing a new RayCluster.

[kevin85421]

@architkulkarni I have already added the details about manual tests.

[kevin85421]

cc @YQ-Wang