Build Headless Service for Multi-Host TPU Worker Pods

[ryanaoleary]

Why are these changes needed?

In order to support Multi-Host TPU worker groups with Kuberay, it's necessary to build a headless service that exposes worker pods to allow pod-to-pod communication. This PR builds a headless service that selects for worker nodes in a RayCluster with NumOfHosts > 1. cc: @richardsliu @kevin85421

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[ryanaoleary]

Following up on this @kevin85421 now that #1913 has been merged.

[kevin85421]

CI has some errors. You should be able to run the unit tests and lint tests locally. Refer to https://github.com/ray-project/kuberay/blob/master/ray-operator/DEVELOPMENT.md for more details.

[kevin85421]

LGTM. Could you
(1) address https://github.com/ray-project/kuberay/pull/1920/files#r1503536623 => [Update] I will open a follow up directly.
(2) explain the details about how you manually tested this PR? Do you check the endpoints attached to the headless service?

Thanks!

[ryanaoleary]

LGTM. Could you (1) address https://github.com/ray-project/kuberay/pull/1920/files#r1503536623 => [Update] I will open a follow up directly. (2) explain the details about how you manually tested this PR? Do you check the endpoints attached to the headless service?

Thanks!

I manually tested by creating a RayCluster with a multi-host worker group and then checking to see if the headless service was created. I then checked to verify that the worker endpoints were exposed by the headless service.

[kevin85421]

Thanks!