Allow ray stop to stop a specific ray start

[joshua-cogliati-inl]

Describe your feature request

The use case is that we have a cluster that is shared by multiple users. We are trying to use ray on this cluster. Since it is shared, to run a program, we request resources (example qsub -l select=4:ncpus=2 ...) and then we start ray on the resources we have been given. Then, when the program is done we need to stop ray.
The issue is, say we request two sets of resources and get the following nodes:
Set1: node1 node2
Set2: node2 node3

Then, when we are done running the program for Set1, we run ray stop on node1 and node2, and that also would kill the ray running on node2 for Set2. That is a problem.

One way to solve this would be if ray stop had some way of specifying which ray to stop, such as by password, or head node and port, or with some other unique identifier.

This was first discussed at:
https://github.com/ray-project/ray/discussions/12103

Note that the slurm documentation ( https://docs.ray.io/en/master/cluster/slurm.html ) for ray suggests: "Clusters managed by Slurm may require that Ray is initialized as a part of the submitted job" which is basically where we run into the problems since if a submitted job has to start things, and you have two jobs, how do you stop the proper ones.

[edoakes]

@richardliaw this seems pretty important for better Slurm support.

[wuisawesome]

I see 3 main ways we could implement this.

1. Take more advantage of OS utilities to kill processes (i.e. pgrep -P $pid_started_by_slurm | xargs kill).

2. Change GCS/raylets/redis process names (everything else should kill itself via fate sharing). Not a big fan of this since it will take many timeouts for this to propagate.

3. Implement an autoscaler node provider which supports namespace isolation (seems like Docker isn't a very popular option in the supercomputing world) and recommend people do that.

[joshua-cogliati-inl]

As a subset of 1. possibly ray start could take a filename and ray start could write the pids that need to be killed into the file, and then ray stop could take the filename and read the pids and kill them. Or if ray start could otherwise communicate a pid out we could grab that and pass it back to ray stop. Or maybe if ray stop can figure out one process that is part of the started, it could use something like SIGUSER1 (or SIGTERM) on it, and then that process can trap the signal and call SIGKILL/SIGTERM on the rest if there is some process that can keep track of the all the ray children.

[wuisawesome]

I believe the ray start process should be the "root" of Ray's process tree on all nodes. Could you try

ray start --head
echo $! > ray_pid.txt

then

pgrep -P $(cat ray_pid.txt) | xargs kill instead of ray stop, or something along those line?

I think we could implement that fairly easily (minus windows support).

[joshua-cogliati-inl]

That didn't work. The Parent ID ends up being init:

UID         PID   PPID  C STIME TTY      STAT   TIME CMD
jjc     155887      1  0 14:26 pts/0    Sl     0:00 <snip>/ray/raylet/raylet <snip>

On the other hand, pgrep -f <redispassword> | xargs kill managed to get everything except redis-server and plasma_store_server since the redispassword is in the command line for rest of the ray programs.

[joshua-cogliati-inl]

Considering that ray stop already grabs the cmdline it could check the redis password and only kill the ones that match, we just need to figure out a way to match the redis-server and plasma_store_server.

[mattip]

Related to #11509

[richardliaw]

(redundant to #11509)