[core][observability] Report idle node information in status and dashboard #39638

vitsai · 2023-09-13T16:53:34Z

Plumbs through idle node information to be reflected in ray status both in the CLI and on the dashboard. Does not include additional changes in the cluster tab of the dashboard UI, but does plumb the status field through to the datasource for dashboard consumption.

List of idle nodes
In verbose mode, will print node activity (reasons node is not idle) for each node

======== Autoscaler status: 2023-09-22 23:08:42.399287 ========
GCS request time: 0.000781s

Node status
---------------------------------------------------------------
Active:
 1 node_328da3b1e9273cf946f6ac3dfee9404dacf429566cd0809a7f04f01c
Idle:
 (no idle nodes)
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Total Usage:
 1.0/36.0 CPU
 0B/36.04GiB memory
 0B/18.02GiB object_store_memory

Total Demands:
 (no resource demands)

Node: 328da3b1e9273cf946f6ac3dfee9404dacf429566cd0809a7f04f01c
 Usage:
  1.0/36.0 CPU
  0B/36.04GiB memory
  0B/18.02GiB object_store_memory
 Activity:
  Resource: CPU currently in use.
  Busy workers on node.

Why are these changes needed?

Related issue number

#35411

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

rickyyx

looks pretty good to me! some nits and comments.

So i guess the actual printing or showing on the dashboard part will be in another PR?

dashboard/datacenter.py

src/ray/gcs/gcs_server/gcs_resource_manager.cc

src/ray/protobuf/autoscaler.proto

src/ray/protobuf/gcs.proto

src/ray/gcs/gcs_server/gcs_resource_manager.cc

src/ray/raylet/scheduling/local_resource_manager.cc

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

rickyyx

Looks pretty good to me!

The only thing I wanna double check is just the Snapshot status setting logic (if the idle one will get overridden), and if we could have a test that tests the new node activity thing e2e? I think so far it's either being mocked. If e2e is non trivial, then some manual tests with ray status -v is also fine, and some unit testing.

And a couple of nits, including

Write in the PR description for what's being changed here in terms of output?

dashboard/datacenter.py

python/ray/autoscaler/_private/autoscaler.py

python/ray/autoscaler/_private/util.py

python/ray/autoscaler/v2/tests/test_utils.py

python/ray/core/generated/autoscaler_pb2.py

python/ray/tests/test_resource_demand_scheduler.py

src/ray/gcs/gcs_client/accessor.cc

src/ray/gcs/gcs_server/gcs_resource_manager.cc

src/ray/raylet/scheduling/local_resource_manager.cc

rickyyx · 2023-09-22T17:03:28Z

Also - i guess the PR doesn't update the dashboard view yet? Or it's automatically handled with changes in the PR?

i.e. the active status:

jjyao

Could you add the PR description?

Ideally also show some example outputs of status and dashboard after your change.

python/ray/autoscaler/_private/autoscaler.py

vitsai · 2023-09-22T17:21:16Z

Right now, the idle state is reflected in dashboard here. I wasn't sure about adding it to that part of the cluster tab because it displays information at a per-worker granularity, whereas we have idle information at a per-node granularity.

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

rickyyx

Is the idle time thing being done outside of this PR? If so, let's document it in a TODO.

dashboard/datacenter.py

python/ray/autoscaler/v2/tests/test_utils.py

src/ray/protobuf/autoscaler.proto

rickyyx · 2023-09-23T01:24:33Z

src/ray/raylet/scheduling/local_resource_manager.cc

  for (const auto &iter : last_idle_times_) {
    if (iter.second == absl::nullopt) {
      // If it is a WorkFootprint
      if (iter.first.index() == 0) {
        switch (std::get<WorkFootprint>(iter.first)) {
        case WorkFootprint::NODE_WORKERS:
-          node_activity << " Node currently has leased workers." << std::endl;
+          resources_data.add_node_activity("Busy workers on node.");


Suggested change

resources_data.add_node_activity("Busy workers on node.");

resources_data.add_node_activity("Active workers.");

?

src/ray/raylet/scheduling/local_resource_manager.cc

rickyyx · 2023-09-23T17:39:34Z

This seems relevant: https://buildkite.com/ray-project/oss-ci-build-pr/builds/36548#018abf99-2bb8-4752-bf9d-98ba0b2b0208

rickyyx

via GIPHY

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

…board (ray-project#39638) --------- Signed-off-by: vitsai <vitsai@cs.stanford.edu>

…and dashboard (ray-project#39638)" This reverts commit a2dedf1.

…board (ray-project#39638) --------- Signed-off-by: vitsai <vitsai@cs.stanford.edu> Signed-off-by: Victor <vctr.y.m@example.com>

rickyyx reviewed Sep 13, 2023

View reviewed changes

vitsai added 3 commits September 16, 2023 00:29

wip

9230582

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

comments

c81d2d8

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

comments

30b41e8

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

vitsai force-pushed the idle-state branch from 7682042 to 30b41e8 Compare September 19, 2023 19:43

vitsai added 6 commits September 21, 2023 05:20

debug

9b64f4f

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

debug

80f7473

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

test fixes new batch

6751079

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

make dashboard work

8d2f9cf

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

more fixes

633e2e8

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

fix dashboard test

bc906c2

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

vitsai marked this pull request as ready for review September 22, 2023 11:05

vitsai requested review from a team, wuisawesome, DmitriGekhtman, ericl, architkulkarni, AmeerHajAli, robertnishihara, pcmoritz and raulchen as code owners September 22, 2023 11:05

vitsai assigned rickyyx Sep 22, 2023

rickyyx reviewed Sep 22, 2023

View reviewed changes

jjyao reviewed Sep 22, 2023

View reviewed changes

python/ray/autoscaler/_private/autoscaler.py Outdated Show resolved Hide resolved

python/ray/autoscaler/_private/autoscaler.py Show resolved Hide resolved

vitsai added 2 commits September 22, 2023 22:53

comments

591fe47

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

change tests too

2633a8a

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

rickyyx reviewed Sep 23, 2023

View reviewed changes

rickyyx approved these changes Sep 23, 2023

View reviewed changes

vitsai added 2 commits September 25, 2023 04:56

fix windows tests

f1f035c

Signed-off-by: vitsai <vitsai@cs.stanford.edu>

Merge branch 'master' into idle-state

2112d54

rickyyx merged commit a2dedf1 into ray-project:master Sep 25, 2023
103 of 107 checks passed

simonsays1980 pushed a commit to simonsays1980/ray that referenced this pull request Sep 26, 2023

[core][observability] Report idle node information in status and dash…

1ca9dc4

…board (ray-project#39638) --------- Signed-off-by: vitsai <vitsai@cs.stanford.edu>

rkooo567 pushed a commit to rkooo567/ray that referenced this pull request Sep 28, 2023

Revert "[core][observability] Report idle node information in status …

49771bb

…and dashboard (ray-project#39638)" This reverts commit a2dedf1.

rkooo567 mentioned this pull request Sep 28, 2023

Revert "[core][observability] Report idle node information in status … #39934

Closed

8 tasks

vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023

[core][observability] Report idle node information in status and dash…

41fd885

…board (ray-project#39638) --------- Signed-off-by: vitsai <vitsai@cs.stanford.edu> Signed-off-by: Victor <vctr.y.m@example.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core][observability] Report idle node information in status and dashboard #39638

[core][observability] Report idle node information in status and dashboard #39638

vitsai commented Sep 13, 2023 •

edited

Loading

rickyyx left a comment

rickyyx left a comment

rickyyx commented Sep 22, 2023

jjyao left a comment

vitsai commented Sep 22, 2023

rickyyx left a comment

rickyyx Sep 23, 2023

rickyyx commented Sep 23, 2023

rickyyx left a comment

	resources_data.add_node_activity("Busy workers on node.");
	resources_data.add_node_activity("Active workers.");

[core][observability] Report idle node information in status and dashboard #39638

[core][observability] Report idle node information in status and dashboard #39638

Conversation

vitsai commented Sep 13, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

rickyyx left a comment

Choose a reason for hiding this comment

rickyyx left a comment

Choose a reason for hiding this comment

rickyyx commented Sep 22, 2023

jjyao left a comment

Choose a reason for hiding this comment

vitsai commented Sep 22, 2023

rickyyx left a comment

Choose a reason for hiding this comment

rickyyx Sep 23, 2023

Choose a reason for hiding this comment

rickyyx commented Sep 23, 2023

rickyyx left a comment

Choose a reason for hiding this comment

vitsai commented Sep 13, 2023 •

edited

Loading