[core] Use the correct way to check whether an actor task is running or not

[kevin85421]

Why are these changes needed?

Ray core worker processes change the process title by calling setproctitle before and after executing tasks. Then, verify_tasks_running_or_terminated uses the following logic to check whether the Ray actor task is running or not:

if psutil.pid_exists(pid) and task_name in psutil.Process(pid).name():

However, the logic is not correct in some cases. To understand the issue, we need to understand the behavior of setproctitle and psutil.Process(pid).name().

• setproctitle:

  • It calls functions like prctl based on the OS and arch under the hood. It updates /proc/pid/task/tid/comm of the calling thread.
  • If the thread is the leader thread (i.e. main thread in Python), /proc/pid/comm will also be updated.
  • /proc/pid/cmdline and /proc/pid/task/tid/cmdline will always be updated based on my observation.
  • Note that comm has a length limitation (15 chars)

• psutil.Process(pid).name()

  • https://github.com/giampaolo/psutil/blob/a17550784b0d3175da01cdb02cee1bc6b61637dc/psutil/__init__.py#L664-L693
  • Try to retrieve /proc/pid/comm.
  • If /proc/pid/comm is truncated, attempt to retrieve the first element of /proc/pid/cmdline; otherwise, return /proc/pid/comm.
  • If cmdline[0] begins with the contents of /proc/pid/comm, return cmdline[0]; otherwise, return the truncated /proc/pid/comm.

If the actor task is not running on the main thread (in which case setproctitle will not update /proc/pid/comm), and if users pass an options.name to the actor that doesn't start with /proc/pid/comm, then task_name in psutil.Process(pid).name() will evaluate to false, which is incorrect.

def test_default_thread_options_name_long_comm_2(self, ray_start_regular):
    """
    `/proc/PID/comm` is truncated to 15 characters, so the process title
    is "ray::VeryLongCo". `psutil.Process.name()` doesn't return `cmdline()[0]`
    because `cmdline()[0]` doesn't start with "ray::VeryLongCo". This is the
    implementation detail of `psutil`.
    """
    task_name = "hello"

    @ray.remote(concurrency_groups={"io": 1})
    class VeryLongCommActor:
        def check_long_comm(self):
            pid = os.getpid()
            assert _is_actor_task_running(pid, task_name)
            # The first 15 characters of "ray::VeryLongCommActor"
            assert psutil.Process(pid).name() == "ray::VeryLongCo"
            assert psutil.Process(pid).cmdline()[0] == f"ray::{task_name}"
            return pid

    a = VeryLongCommActor.remote()
    pid = ray.get(a.check_long_comm.options(name=task_name).remote())
    wait_for_condition(lambda: not _is_actor_task_running(pid, task_name))

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[kevin85421]

This is to fix #51058 (comment).