[RayService][Hotfix] Hotfix for Flaky Zero Downtime Rollout Test

[Yicheng-Lu-llll]

Why are these changes needed?

The Zero Downtime Rollout Test is flaky for the following reason:

The current logic for the Zero Downtime Rollout operates as follows:

1. Once the pending RayCluster is ready for service, the RayService Controller updates its status to 'running' and updates the selection label for the serve service.
2. The RayService Controller then waits for 60 seconds before deleting the old RayCluster. This 60-second interval allows time for the serve service to be updated.

However, a critical issue is that the 'running' status merely indicates the readiness of the new RayCluster to serve traffic; it does not confirm that the serve service has begun redirecting traffic to the new RayCluster. As a result, requests sent after the status change to 'running' might still be responded to by the old RayCluster.

This scenario may lead to high-probability assertion errors in RayService CI tests, as responses expected from the new RayCluster may still coming from the old RayCluster. Some examples can be found here:

• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/2891#018cfaca-c122-4c5a-a28d-6a8a6503ef2f/251-2146
• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/2942#018d187d-3036-45b0-8494-5176a7755358/251-2097
• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/2942#018d18b8-947e-4150-97fe-2cb85760f939/251-2094
• Adding serve support for Api server #1728 (comment)

To mitigate this, and before refactoring the Rollout logic and redefine the of status, this PR adds delay to ensure the serve service is ready to redirect traffic to the new RayCluster.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

Some failures in this build appear to be related to this issue. @Yicheng-Lu-llll would you mind confirming it? Thanks!

[Yicheng-Lu-llll]

Yes, the CI failures are due to the issue mentioned here. I've included them in the sample section of my PR description. Thank you!

[kevin85421]

I haven't succeeded in reproducing the issue on my laptop. However, I encountered the issue twice when I ran the test three times in this build. After I add the hotfix to my PR, it passes the first time. Although the data points are limited, this may still somewhat validate the assumptions in this PR.