[Core] Add a utility to check GCS / Ray cluster health

[mwtian]

Why are these changes needed?

• Provide a utility to test connecting to a Ray cluster and verify it has the same Ray version. This is useful to check if a Ray cluster is available at a given address, without connecting to the cluster with the more heavyweight ray.init(). This utility is integrated with ray memory to provide a better error message when the Ray cluster is unavailable. There seem to be user demand for exposing this as an API as well.
• Improve the error message when the address provided to Ray does not contain port.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

Just curious, but how would this be different from

ray/dashboard/head.py

Line 60 in f91a134

request = gcs_service_pb2.CheckAliveRequest()

?

[mwtian]

They look very similar. The only difference is that ping_cluster() has the additional check for Python and Ray version compatibility. Should we update dashboard to use this utility or use CheckAliveRequest() here instead?

[rkooo567]

They look very similar. The only difference is that ping_cluster() has the additional check for Python and Ray version compatibility. Should we update dashboard to use this utility or use CheckAliveRequest() here instead?

I don't have a strong opinion what to use, but I think we should unify to one (seems like both are for health checking the cluster).

cc @wuisawesome do you see any downsides of just using this util?

[wuisawesome]

Please please add this functionality to ray healthcheck (which uses CheckAliveRequest). We shouldn't let this type of functionality deviate between components

[mwtian]

How about using CheckAliveRequest() to check connectivity, and use cluster metadata to check Python & Ray versions? I'm not sure we want to add unnecessary data to the health check response.

[mwtian]

I thought about the use case and only verifying Ray versions match should be sufficient.

[wuisawesome]

Ok can we change the api name to make it clear that this is not a healthcheck or way to ping the cluster? some form of cluster metadata that we can expose through the dashboard api seems reasonable.

[mwtian]

@wuisawesome which API are you referring to, ping_cluster() or CheckAlive?

[rkooo567]

I will leave it up to @wuisawesome also cc @jon-chuang

[wuisawesome]

@wuisawesome which API are you referring to, ping_cluster() or CheckAlive?

CheckAlive is a grpc health check, and powers ray health-check. (I don't think the grpc endpoint is stable across versions since only ray health-check is).

If you're trying to add a cluster metadata api and not a health check, I'm saying you should name it GetClusterMetadata. Calling it ping is extremely confusing since well known libraries like Redis use ping as a health check.

Btw it's also still not clear to me why you don't just add this in the health check. The main reason to separate them would be if we have some big metadata/expensive query that we didn't want to run in a health check (which we don't).

[mwtian]

Btw it's also still not clear to me why you don't just add this in the health check. The main reason to separate them would be if we have some big metadata/expensive query that we didn't want to run in a health check (which we don't).

Actually I'm using the CheckAlive rpc right now. Renaming the function to health check.

[mwtian]

The plan is to add a public health check API in a subsequent PR.