[core] The New GcsClient binding #46186
Conversation
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
rynewang
requested review from
scottjlee,
bveeramani,
raulchen,
stephanie-wang,
omatthew98 and
a team
as code owners
| @@ -661,6 +662,8 @@ def test_get_applications_while_gcs_down( | |||
| ): | |||
| # Test serve REST API availability when the GCS is down. | |||
| monkeypatch.setenv("RAY_SERVE_KV_TIMEOUT_S", "3") | |||
| importlib.reload(ray.serve._private.constants) # to reload the constants set above | |||
There was a problem hiding this comment.
why do we need to change this?
There was a problem hiding this comment.
By default, Ray Serve uses infinite timeout for internal kv put/get. To test when GCS is down, it sets the timeout to 3s. However this setting never worked because the env did not load.
Previous PythonGcsClient would return error of GrpcUnavailable on GCS down even if timeout is inf. The new GcsClient properly infinitely retries and hangs. To make the env work, we need to reload it.
python/ray/includes/common.pxi
Outdated
|
|
||
| cdef class GcsClientOptions: | ||
| """Cython wrapper class of C++ `ray::gcs::GcsClientOptions`.""" | ||
| cdef: | ||
| unique_ptr[CGcsClientOptions] inner | ||
|
|
||
| @classmethod | ||
| def from_gcs_address(cls, gcs_address): | ||
| def from_gcs_address(cls, gcs_address, cluster_id_hex=None): |
There was a problem hiding this comment.
this method name is no longer accurate
There was a problem hiding this comment.
Also moving cluster_id into GcsClientOptions can be its own PR?
| @@ -104,7 +104,7 @@ def ping(self): | |||
|
|
|||
| gcs_client = GcsClient(address=ray.get_runtime_context().gcs_address) | |||
|
|
|||
| with pytest.raises(ray.exceptions.RpcError): | |||
| with pytest.raises(ray.exceptions.RaySystemError): | |||
There was a problem hiding this comment.
is there any breaking changes. Ideally this PR shouldn't touch serve code or any tests.
There was a problem hiding this comment.
This is a fix forward. drain_node returns an error from GCS side which should not be considered a RpcError (indicates network issue). Ray serve changes fix bad test fixtures.
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Creates a direct Cython binding for
ray::gcs::GcsClientand replaces the existingPythonGcsClientbinding. The new binding is enabled by default; one can switch back withRAY_USE_OLD_GCS_CLIENT=1.The new binding is in its own file
gcs_client.pxiincluded by_raylet.pyx.Changes:
cluster_idfrom arg to a GcsClientOptions field.timeout_msarg forNodeInfoAccessor::AsyncGetAllandJobInfoAccessor::AsyncGetAll.JobInfoAccessor::GetAllandNodeInfoAccessor::DrainNodesandNodeResourceInfoAccessor::GetAllResourceUsage.NodeInfoAccessor::GetAllNoCache.python/ray/tune/tests/test_tune_restore.py::ResourceExhaustedTest::test_resource_exhausted_info