[Core][Cluster] Actors do not trigger the scale up process because the default num_cpus seems not work

[orcahmlee]

What happened + What you expected to happen

I set up a Ray cluster on K8s which would auto-scaling the ray workers from 0 to 6. The values.yaml as follows:

image: rayproject/ray:1.12.0-py38
upscalingSpeed: 1.0
idleTimeoutMinutes: 5
headPodType: rayHeadType
podTypes:
  rayHeadType:
    CPU: 3
    memory: 28Gi
    rayResources: { "CPU": 0 }
  rayWorkerType:
    minWorkers: 0
    maxWorkers: 6
    memory: 28Gi
    CPU: 3
    GPU: 0

---

@ray.remote
def go():
    sleep(1)
    return "OK"


objs = [go.remote() for _ in range(50)]

I submitted 50 tasks when there has 0 ray worker. The ray status showed

Demands:
{'CPU': 1.0}: 50+ pending tasks/actors

Then the scaling up process would be executed immediately. This is the expected behavior.

---

@ray.remote
class ThreadedActor:
    def task(self):
        return f"native id: {threading.get_native_id()}"

actors = [ThreadedActor.options(max_concurrency=5).remote() for _ in range(50)]

However, I submitted 50 actors when there has 0 ray worker. The scaling up process would NOT be executed, and the ray status showed:

Demands:
{}: 50+ pending tasks/actors

I noticed the ray staus message did not list the CPU numbers, so I did two trials.

@ray.remote(num_cpus=1)  # <--- modified
class ThreadedActor:
    def task(self):
        return f"native id: {threading.get_native_id()}"

actors = [
    ThreadedActor.options(max_concurrency=5).remote() for _ in range(50)
]

@ray.remote
class ThreadedActor:
    def task(self):
        return f"native id: {threading.get_native_id()}"

actors = [
    ThreadedActor.options(num_cpus=1, max_concurrency=5).remote() for _ in range(50)  # <--- modified
]

The scaling up process would be executed immediately, when I added num_cpus=1 either to the decorator or the options(). The ray status showed

Demands:
{'CPU': 1.0}: 50+ pending tasks/actors

It's weird the default parameter seems not to affect Actors.

Versions / Dependencies

• Python 3.8.13
• ray 1.12.0
• GKE 1.21.12-gke.1700

Reproduction script

from time import sleep
import threading
import ray


@ray.remote
def go():
    sleep(1)
    return "OK"


@ray.remote
class ThreadedActor:
    def task(self):
        return f"native id: {threading.get_native_id()}"


if __name__ == "__main__":
    ray.init(address="ray://HEAD-IP:10001")

    objs = [go.remote() for _ in range(50)]  # scaling up works

    actors = [
        ThreadedActor.options(max_concurrency=5).remote() for _ in range(50)  # scaling up does not work
    ]

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[DmitriGekhtman]

This is a subtle issue that has been resolved in master in
#26813

@wuisawesome can elaborate here on why actor scheduling might not trigger scale-up with a 0-CPU head node

[wuisawesome]

@orcahmlee happy to elaborate if you're interested, but for now i will close this as a duplicate of #26806. Please reopen if you can repro on a nightly after that commit.

[orcahmlee]

Thanks to both of you, I'll check the nightly version.