[Bug] RayService with GCS FT HA issue

[kevin85421]

Why are these changes needed?

According to the comment in issue #1463, only the Ray head and workers with Serve replicas will have an HTTPProxyActor in a stable state. Without this PR, if the replica is scheduled on the head, there will be only one replica in the cluster and the worker will not have an HTTPProxyActor. Hence, the K8s serve service has only one endpoint from the head, and this endpoint will be removed when the head Pod is terminated, and thus there will be no HTTPProxyActor & replica available.

This PR makes sure both the head Pod and worker Pod have 1 replica and 1 HTTPProxyActor. Hence, when the head Pod is deleted, the K8s serve service still has an endpoint from worker to achieve high-availability.

Related issue number

Closes #1463

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

I made some changes based on @YQ-Wang's reproduction script.

• Step 1: Create a GCS FT-enabled RayService with this gist.

  • I changed the number of replica from 1 to 2, and each one requires 1 CPU.

    deployments:
      - name: ImageClassifier
        num_replicas: 2
        ray_actor_options:
          num_cpus: 1

• Step 2: Create a Ray Pod by this gist, and run the following script to send requests to the RayService.

  import requests
  
  # TODO: Change this to your image path
  image_path = "./dog.png"
  
  url = "http://rayservice-mobilenet-serve-svc:8000"
  index = 0
  fail = 0
  
  while True:
      print(f"iter : {index}, fail: {fail}")
      files = {"image": open(image_path, "rb")}
      try:
          response = requests.post(url, files=files)
      except Exception as e:
          print(e)
          fail += 1
      print(response.text)
      index += 1

• Step 3: Kill the head Pod. Typically, only the 0 to 2 requests that were being processed by the Ray head at that moment will be dropped. Users should implement application-level retry to avoid failure.
  

[kevin85421]

cc @YQ-Wang

[architkulkarni]

How did this YAML even work before without a deployments field?

[kevin85421]

The only 1 replica will be deployed on the head Pod, as KubeRay submits the Serve application once the head node is ready. In most cases, workers are not ready.

[architkulkarni]

The PR description and fix makes sense to me.

By the way, in general, if the user specifies N>1 replicas, do they need to do any additional work to prevent the case that all N are scheduled on the head node? My guess is no, Ray Serve should automatically spread them out across the worker nodes.

In any case, we should probably add in the Serve HA documentation that num_replicas should be set to something greater than 1. (It sounds basic but it's still better to say it explicitly)

[kevin85421]

By the way, in general, if the user specifies N>1 replicas, do they need to do any additional work to prevent the case that all N are scheduled on the head node? My guess is no, Ray Serve should automatically spread them out across the worker nodes.

If I understand correctly, unfortunately, users must set the relevant scheduling configurations (e.g., num_cpus) to ensure that more than one Pod contains the deployment's replica. We also track the progress in #1492.

In any case, we should probably add in the Serve HA documentation that num_replicas should be set to something greater than 1. (It sounds basic but it's still better to say it explicitly)

I will add a section for high-availability in the RayService document.