[RayService] Fixed issue where the custom serve port is not reflected in the serve health check for worker Pods

[Yicheng-Lu-llll]

Why are these changes needed?

As title.

ray-service.different-port.yaml is used to demonstrate how to set custom serve port. However, the instruction is not complete.

when running ray-service.different-port.yaml, it may appear that everything is functioning correctly. However, the custom serve port is not accurately reflected in the serve health check for worker Pods. The absence of errors is due to the serve app only requiring a minimal amount of CPU resources. Consequently, the worker Pod might not even host a serve replica.

To illustrate this issue, I have created a Ray service similar to ray-service.different-port.yaml, but with a modified serve app configuration ensuring each Pod has one serve replica. As shown in the experiment below, the worker Pod consistently fails the health check because it defaults to using the serve port value of 8000.

kubectl apply -f https://raw.githubusercontent.com/Yicheng-Lu-llll/serve-file/main/ray-service.different-port.bad.example.yaml

# The ray.io/serve label is used to indicate whether a Pod passes the serve health check.
kubectl describe $(kubectl get pods -o=name | grep worker) | grep  "ray.io/serve"
#ray.io/serve=false

This issue occurs because Kuberay determines the serve port by examining the ray container. If the serve port is not set in the worker configuration, it defaults to port 8000.

btw, I am not sure if it is a good idea to change every Rayservice samples to enforce every Ray Pod at least has one replica. This kind of error may occur in future.

Related issue number

Checks

# Add a custom serve port to both head pod and worker Pod.
kubectl apply -f https://raw.githubusercontent.com/Yicheng-Lu-llll/serve-file/main/ray-service.different-port.yaml

# The ray.io/serve label is used to indicate whether a Pod passes the serve health check.
kubectl describe $(kubectl get pods -o=name | grep worker) | grep  "ray.io/serve"
#ray.io/serve=true

[kevin85421]

LGTM! Would you mind verifying if the RayService documentation in the Ray repository instructs users to add the serve port to both head and worker Pods?