Fix in HeadPod Service Generation logic which was causing frequent reconciliation

[msumitjain]

Why are these changes needed?

BuildServiceForHeadPod method creates the corev1.Service object without ensuring the order of ports. When the rayservice_controller reconciliation loop updates the Service Spec, K8s detects a change in the and applies it. This will lead to a performance degradation.

Related issue number

Closes 1044

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

LGTM! Thank you for the contribution.

I cloned the fork and tested it by the following instructions:

Reproduce

helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0
# path: ray-operator/config/samples
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Wait for 5 mins
# (a) Check the order of ports
kubectl describe svc rayservice-sample-head-svc

# (b) Check resourceVersion
kubectl get svc rayservice-sample-head-svc -o=jsonpath='{.metadata.resourceVersion}'

# Wait for another 5 mins
# (c) Check the order of ports
kubectl describe svc rayservice-sample-head-svc

# (d) Check resourceVersion
kubectl get svc rayservice-sample-head-svc -o=jsonpath='{.metadata.resourceVersion}'

# (c) is possible to be not equal to (a), and (d) must be different from (b).

Test this PR

# Build Docker image (path: ray-operator)
make docker-image
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0 --set image.repository=controller,image.tag=latest
# path: ray-operator/config/samples
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Wait for 5 mins
# (a) Check the order of ports
kubectl describe svc rayservice-sample-head-svc

# (b) Check resourceVersion
kubectl get svc rayservice-sample-head-svc -o=jsonpath='{.metadata.resourceVersion}'

# Wait for another 5 mins
# (c) Check the order of ports
kubectl describe svc rayservice-sample-head-svc

# (d) Check resourceVersion
kubectl get svc rayservice-sample-head-svc -o=jsonpath='{.metadata.resourceVersion}'

# (c) should be equal to (a), and (d) should be equal to (b).

This PR prevents the RayService's head service from being updated by Kubernetes APIServer. However, we still need to make sure we only call the Update function if the object has actually been modified to avoid unnecessary API calls and potential race conditions (i.e. 409 conflict) (At this moment, we call Update() in every reconciliation (link)). This follow-up will be tracked by #983.

[kevin85421]

Hi @msumitjain, would you mind rebasing with the master branch? Some CI-related PRs are merged recently. Thanks!