Global State Available Resources Hangs on Node Removal

[richardliaw]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 16
• Ray installed from (source or binary): Source
• Ray version: master
• Python version: 3.6.6
• Exact command to reproduce:

ray.global_state.available_resources() on a small cluster

Describe the problem

In [8]: ray.global_state.available_resources()

hangs after a node is removed.

Source code / logs

...
  'NodeManagerAddress': '169.229.49.172',
  'NodeManagerPort': 38033,
  'ObjectManagerPort': 42375,
  'ObjectStoreSocketName': '/tmp/plasma_store65771381',
  'RayletSocketName': '/tmp/raylet8912751',
  'Resources': {'GPU': 0.0, 'CPU': 2.0}},
 {'ClientID': 'ed085bb78046ccbc423fb6587ab1fcbf838514da',
  'IsInsertion': True,
  'NodeManagerAddress': '169.229.49.173',
  'NodeManagerPort': 32821,
  'ObjectManagerPort': 34913,
  'ObjectStoreSocketName': '/tmp/plasma_store60872177',
  'RayletSocketName': '/tmp/raylet24528210',
  'Resources': {'GPU': 0.0, 'CPU': 2.0}}]

In [7]: The node with client ID ed085bb78046ccbc423fb6587ab1fcbf838514da has been marked dead because the monitor has missed too many heartbeats from it.
In [7]: ray.global_state.client_table()
Out[7]:
[{'ClientID': 'c9f0b409ca553018762fadba26de1dffa4338478',
  'IsInsertion': True,
  'NodeManagerAddress': '169.229.49.172',
  'NodeManagerPort': 38033,
  'ObjectManagerPort': 42375,
  'ObjectStoreSocketName': '/tmp/plasma_store65771381',
  'RayletSocketName': '/tmp/raylet8912751',
  'Resources': {'GPU': 0.0, 'CPU': 2.0}},
 {'ClientID': 'ed085bb78046ccbc423fb6587ab1fcbf838514da',
  'IsInsertion': True,
  'NodeManagerAddress': '169.229.49.173',
  'NodeManagerPort': 32821,
  'ObjectManagerPort': 34913,
  'ObjectStoreSocketName': '/tmp/plasma_store60872177',
  'RayletSocketName': '/tmp/raylet24528210',
  'Resources': {'GPU': 0.0, 'CPU': 2.0}},
 {'ClientID': 'ed085bb78046ccbc423fb6587ab1fcbf838514da',
  'IsInsertion': False,
  'NodeManagerAddress': '',
  'NodeManagerPort': 0,
  'ObjectManagerPort': 0,
  'ObjectStoreSocketName': '',
  'RayletSocketName': '',
  'Resources': {}}]

In [8]: ray.global_state.available_resources()
...

@pschafhalter

[richardliaw]

Hangs on:

KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-8-94b41eaa45df> in <module>()
----> 1 ray.global_state.available_resources()

/data/rliaw/ray/python/ray/experimental/state.py in available_resources(self)
   1394                 client_ids = {
   1395                     client["ClientID"]
-> 1396                     for client in self.client_table()
   1397                 }
   1398

/data/rliaw/ray/python/ray/experimental/state.py in client_table(self)
    502             message = self.redis_client.execute_command(
    503                 "RAY.TABLE_LOOKUP", ray.gcs_utils.TablePrefix.CLIENT, "",
--> 504                 NIL_CLIENT_ID)
    505
    506             # Handle the case where no clients are returned. This should only

/data/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/redis/client.py in execute_command(self, *args, **options)
    665         connection = pool.get_connection(command_name, **options)
    666         try:
--> 667             connection.send_command(*args)
    668             return self.parse_response(connection, command_name, **options)
    669         except (ConnectionError, TimeoutError) as e:

[robertnishihara]

I think the correct solution is to fix it by doing #2852 and then available_resources should ignore dead nodes.

[richardliaw]

This is fixed in #3004.