[Core] Better propogate node death information

[ruisearch42]

Why are these changes needed?

This is the first PR of a series to better propagate and expose node death information.

Background: in Ray, a node can be either ALIVE or DEAD, and the death reason of a node could be unexpected (e.g., crash) or expected (e.g., idle termination or spot preemption). Currently, GCS knows the actual death reason, but this information is not shown to the users. As a result, users might think a node crashes when they see it is DEAD, while in reality these could be expected scenarios such as spot preemption. This may give users the wrong impression that Ray is unstable. In the series of changes, we are going to better propagate and expose the node death info.

This PR does the following:

• Add reason_message to NodeDeathInfo so that we track more verbose node death info
• Introduce UnregisterNode PRC in GCS and use it instead of DrainNode RPC wherever appropriate

This PR is the basis for future changes where we:

• Maintain and expose the proper node death information for cases where cloud provider forcefully kill the instance before node draining is complete
• Expose node death information to dashboard as well as exception/error messages

See more details in design doc:
https://docs.google.com/document/d/1tn6Uj-SoEAaBu5_HWl4dNo3JBzTd_w09RPsIWQpUU_s/edit

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jjyao]

LG

[jjyao]

Let's add some tests

[jjyao]

lgtm

[jjyao]

[2024-05-14T22:35:24Z] Traceback (most recent call last):
[2024-05-14T22:35:24Z]   File "ci/lint/pytest_checker.py", line 126, in <module>
[2024-05-14T22:35:24Z]     main(data)
[2024-05-14T22:35:24Z]   File "ci/lint/pytest_checker.py", line 111, in main
[2024-05-14T22:35:24Z]     raise RuntimeError(
[2024-05-14T22:35:24Z] RuntimeError: Found py_test files without `if __name__ == "__main__":` snippet:
[2024-05-14T22:35:24Z] python/ray/tests/test_node_death.py
[2024-05-14T22:35:24Z] If this is intentional, please add a `no_main` tag to bazel BUILD entry for those files.