[ray] worker_start_ray_commands are not executed for private cluster

[neychev]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 16.04
• Ray installed from (source or binary): pip
• Ray version: 0.7.0
• Python version: 3.6.7
• Exact command to reproduce:

Describe the problem

I am following private cluster setup instructions, but only head node starts. Few interesting points:

• Seems similar to issue [autoscaler] Workers can silently fail to start. #3408
• Adding initialization_commands: [] fixes the KeyError mentioned in [autoscaler] KeyError when starting private cluster #4559

Source code / logs

cluster_name: tesq_cluster
min_workers: 48
max_workers: 48
initial_workers: 48
provider:
    type: local
    head_ip: ip1
    worker_ips: [ip2, ip3, ip4]
auth:
    ssh_user: tesq
    ssh_private_key: /home/me/.ssh/keys/local_user
file_mounts: {}
setup_commands: []
initialization_commands: []
head_setup_commands: []
worker_setup_commands: []

head_start_ray_commands:
    - source activate py3_prod && ray stop
    - echo 'I am here' >> /home/tesq/new_file.txt
    - source activate py3_prod && ulimit -c unlimited && ray start --head --redis-port=6379
worker_start_ray_commands:
    - echo 'I am there' >> /home/tesq/new_file.txt
    - source activate py3_prod && ray stop
    - echo 'I am there' >> /home/tesq/new_file.txt
    - source activate py3_prod && ray start --redis-address=ip1:6379

After that only head node starts, and only on the head node I see the created file new_file.txt
Example output of command ray.global_state.client_table()

{'ClientID': 'a7ce937ffcbece9b25a779fa126ba47edef27267',
  'IsInsertion': True,
  'NodeManagerAddress': 'ip1',
  'NodeManagerPort': 45759,
  'ObjectManagerPort': 34107,
  'ObjectStoreSocketName': '/tmp/ray/session_2019-05-30_15-51-46_16481/sockets/plasma_store',
  'RayletSocketName': '/tmp/ray/session_2019-05-30_15-51-46_16481/sockets/raylet',
  'Resources': {'GPU': 3.0, 'CPU': 24.0}},

Update:
Seems very similar to issue #3190
But files monitor.err and monitor.out are empty.

[neychev]

@robertnishihara Sorry for summoning you exactly, but this issue seems very similar to your #3190

[neychev]

Still present on 0.7.1. ray up simply does not execute the worker_start_ray_commands
For others seeking for the solution: the old ray 0.5.3 approach still can be used (despite I prefer using ssh in a loop instead of parallel-ssh). Link: https://ray.readthedocs.io/en/ray-0.5.3/using-ray-on-a-large-cluster.html

[neychev]

cc @jovany-wang

[raulchen]

Did you see any exceptions under /tmp/ray/session_xxx/logs/raylet.err or redis.err or plasma_store.err?
And what's the output of running ray start --redis-address=ip1:6379 on the worker node?

[neychev]

@raulchen as far as I can see, the /tmp/ray/session_xxx/ directory is created when some tasks are being executed using the ray cluster. Just on ray up no directory is created in my case.

I start the ray process on the worker nodes just like that via ssh: ray start --redis-address=ip1:6379. No errors, everything works as supposed. No errors during the tasks execution as well.

According to the fact that no file is generated on the worker nodes with the - echo 'I am there' >> /home/tesq/new_file.txt command, it seems that the worker_start_ray_commands from .yaml file are just not executed.

[sp608]

@neychev were you able to figure this out?

[richardliaw]

@sp608 I just tried this out, and this should work now; can you try the nightlies/latest master? You should install this on all nodes (put it in setup_commands as pip install -U whl).

https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master

[dclong]

I have encounter a similar issue. Neither the setup_commands nor the worker_start_ray_commands is executed. Below is the information of my environment.

OS: Ubuntu 18.04
Python: 3.6.8
Ray: 0.8.2

And my configuration for the autoscaler to start a private cluster.

min_workers: 8
initial_workers: 8
max_workers: 8

provider:
    type: local
    head_ip: 10.148.186.178
    worker_ips: [10.148.186.18]
    use_internal_ips: true

auth:
    ssh_user: USER_NAME
    ssh_private_key: ~/.ssh/id_rsa

# Files or directories to copy to the head and worker nodes. 
file_mounts: {
#    "/path1/on/remote/machine": "/path1/on/local/machine",
#    "/path2/on/remote/machine": "/path2/on/local/machine",
}

setup_commands:
    - echo 'Starting Ray ...' > ~/ray.log

# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
    - ray stop
    - ray start --head --redis-port=6379

worker_start_ray_commands:
    - ray stop
    - ray start --address=10.148.186.178:6379

On the head node I can see a file ~/ray.log with the line Starting Ray ... is created while it is not created on the worker node.

[gimzmoe]

ray v0.8.4
python 3.6.9
Ubuntu 18.04.4
I'm running into this same thing, none of the commands (setup_commands) or (worker_start_ray_commands) appear to be executing.

I guess it might not be obvious, but that includes the bit about starting up the worker clients. Basically only the head node is launched, none of the workers appear to be executing any commands "ray start" or otherwise.

[jaromrax]

@sp608 I just tried this out, and this should work now; can you try the nightlies/latest master? You should install this on all nodes (put it in setup_commands as pip install -U whl).

https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master

Same issue, version 0.8.5 doesn't ssh to worker's IPs on ray up cluster.yaml.

I tried 0.9.0.dev0, with the same effect.
https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp36-cp36m-manylinux1_x86_64.whl