[core] Better error message for task/actors when unschedulable (integrate with autoscaler)

[richardliaw]

The problem

This error message is not very actionable nor usable:

2021-05-20 01:48:33,485	WARNING worker.py:1115 -- The actor or task with ID fffffffffffffffffc714cea33cb71acd82c19fb01000000 cannot be scheduled right now. It requires {CPU_group_7e13d05021cb9c6d4284227951fdd56a: 1.000000} for placement, but this node only has remaining {3.000000/4.000000 CPU, 11.200000 GiB/11.200000 GiB memory, 0.000000/1.000000 GPU, 5.000000 GiB/5.000000 GiB object_store_memory, 0.000000/1.000000 CPU_group_7e13d05021cb9c6d4284227951fdd56a, 0.000000/1.000000 CPU_group_1_7e13d05021cb9c6d4284227951fdd56a, 0.000000/1.000000 GPU_group_1_7e13d05021cb9c6d4284227951fdd56a, 0.000000/1.000000 GPU_group_7e13d05021cb9c6d4284227951fdd56a, 1.000000/1.000000 accelerator_type:T4, 1.000000/1.000000 node:172.31.57.235}

A few improvements are needed:

• We should show the specific task/actor in the error message. This includes
  • Knowing whether it is a task or actor
  • Providing the class or function name
  • The call site where the task or actor was launched (nice to have)
• The autoscaler should drive printing of this message. There are two cases:
  • If the autoscaler is actively scaling up to handle the task, we should suppress the message (unless it is taking longer than a certain time).
  • If the autoscaler is not able to scale up, we should print this accordingly from the autoscaler event logs.

[DLWCMD]

I am commenting here as the related issues seem to have been closed. If should should be on another issue, please let me know.

I have received the same blocking message as reported in 13905: The actor or task cannot be scheduled right now. My use case is associated with Poputation Based Training replays: [https://docs.ray.io/en/master/tune/tutorials/tune-advanced-tutorial.html#replaying-a-pbt-run].

In my case, the pbt policy text file specifies 6 workers. As reported by another user, if I manually change that value to 1, the replay runs as expected, albeit slowly since I am using only one worker. Further, the replay will run if local mode is activated.

FYI, I have experienced the same issue with loading and running checkpoints not associated with PBT. A better error message would be useful, particularly if it described how to resolve the problem without giving up the benefits of multiple workers.

I have code and sample PBT policy files that I would be happy to share if you feel they would be useful.

Thanks.

David Wilt

[rkooo567]

@DLWCMD thanks a lot! After handling the first half of TODO, I will ping you if the error message will be useful for your case.

[DLWCMD]

TY. Let me know I can assist. David L. Wilt 3272 Bayou Road Longboat Key FL 34228 ***@***.*** 540-420-0844 From: SangBin Cho ***@***.***> Reply-To: ray-project/ray ***@***.***> Date: Monday, August 9, 2021 at 1:55 PM To: ray-project/ray ***@***.***> Cc: DLWCMD ***@***.***>, Mention ***@***.***> Subject: Re: [ray-project/ray] [core] Better error message for task/actors when unschedulable (integrate with autoscaler) (#15933) @DLWCMD thanks a lot! After handling the first half of TODO, I will ping you if the error message will be useful for your case. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[rkooo567]

#15962 => This will be also done as a part of this work.