Initial support for external Redis and GCS HA #294

wilsonwang371 · 2022-06-08T00:01:43Z

Why are these changes needed?

Initial support for GCS HA in kuberay.

Design document is at: https://github.com/wilsonwang371/kuberay/blob/wilson/ray2.0_gcs_ha/docs/guidance/gcs-ha.md

Changes

added yaml for Ray GCS HA + External Redis Deployment
a document describing Ray GCS HA details.
introduced new annotation in RayCluster to enable Ray GCS HA
Added Event reconcile function so that in case of Readiness Probe failure, we can try to recover head or worker node if possible.
Added Liveness and Readiness Probes based on new features introduced in [dashboard][2/2] Add endpoints to dashboard and dashboard_agent for liveness check of raylet and gcs ray#26408
Added Ray detached actor and Ray serve basic tests in Compatibility Test.
Updated github workflow to reduce its running time

Enable GCS HA

To enable GCS HA, in RayCluster yaml file, a new annotation is required.

apiVersion: ray.io/v1alpha1
kind: RayCluster
metadata:
  annotations:
    ray.io/ha-enabled: "true" # <- add this annotation enable GCS HA
...

When this annotation is added to RayCluster yaml file, all newly created head node and worker nodes pods in this ray cluster will have the same annotation attached.

ray.io/ha-enabled: "true"

Readiness Probe & Liveness Probe

Readiness Probe and Liveness Probe are both used by kuberay to help Ray cluster recover from failures.

Readiness Probe is used to discover early failures and recover if possible. (recovery action is not implemented yet.)
Liveness Probe is used as the last resort to recover the cluster by restarting failed worker/head node

By default, if GCS HA is enabled, default Readiness Probe and default Liveness Probe will be added to the newly created pods in this Ray cluster. If user specified different Readiness Probe & Liveness Probe, default ones will not be added to the head & worker nodes created.

Related issue number

#290

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

wilsonwang371 · 2022-06-08T00:03:17Z

cc @iycheng @Jeffwan @scarlet25151

Jeffwan · 2022-06-08T10:19:16Z

To be clear, does it work for 1.12.1 or just nightly version?

wilsonwang371 · 2022-06-09T23:08:54Z

To be clear, does it work for 1.12.1 or just nightly version?

Right now it is for nightly because @iycheng 's changes are only available in nightly build. It might be available later in 1.13 or later versions. This need confirmation from Yi Cheng's side.

DmitriGekhtman · 2022-07-18T23:55:20Z

@DmitriGekhtman for this.

@brucez-anyscale ,

@DmitriGekhtman for what? :)

docs/guidance/gcs-ha.md

ray-operator/controllers/ray/raycluster_controller.go

docs/guidance/gcs-ha.md

DmitriGekhtman · 2022-07-19T01:58:38Z

ray-operator/controllers/ray/common/pod.go

+					pod.Spec.Containers[rayContainerIndex].LivenessProbe = probe
+				}
+				// add liveness probe exec command in case missing
+				initLivenessProbeHandler(pod.Spec.Containers[rayContainerIndex].LivenessProbe, rayNodeType)


Just a thought for, maybe for later -- I wonder if these probes are useful for restarting pods even if HA isn't enabled.

sounds good. we can have some discussion on this later.

DmitriGekhtman

Looks great. Left a couple of comments on the new docs.

DmitriGekhtman · 2022-07-19T18:41:32Z

To justify the complexity of the readiness probe and event handling, we should, if we haven't yet, plan what kind of recovery logic we want to implement (besides deleting the head pod).

ray-operator/controllers/ray/common/constant.go

DmitriGekhtman · 2022-07-20T00:50:24Z

Are we good to merge this yet?

brucez-anyscale · 2022-07-20T05:27:12Z

I think we can merge this and have followup prs to improve it

Jeffwan · 2022-07-20T17:20:36Z

@DmitriGekhtman @brucez-anyscale waiting for a test fix. I am asking @wilsonwang371 to help fix it and then we can merge. It should be good by this noon

remove duplicated constants Update docs/guidance/gcs-ha.md Co-Authored-By: Dmitri Gekhtman <62982571+DmitriGekhtman@users.noreply.github.com>

wilsonwang371 · 2022-07-20T19:56:40Z

@iycheng it seems like external storage namespace has some issue. I temporarily disabled this, we may need to do some debug later on this.

brucez-anyscale · 2022-07-20T23:35:56Z

@iycheng it seems like external storage namespace has some issue. I temporarily disabled this, we may need to do some debug later on this.

@iycheng @wilsonwang371 Let's follow up and solve it.

wilsonwang371 · 2022-07-24T19:05:00Z

@iycheng it seems like external storage namespace has some issue. I temporarily disabled this, we may need to do some debug later on this.

@iycheng @wilsonwang371 Let's follow up and solve it.

I have posted a new MR. #406

@Jeffwan

* add implementation and test cases for ray ha * bug fix add ray serve test use label to specify ray ha enabled or not * bug fix * update github pipeline * update readiness & liveness command * add Ray GCS HA document * bug fix * address comment from @Jeffwan * use annotation for ray gcs ha * remove redis install command * typo fix and create function initTemplateAnnotations * fix probe initial parameter issue * support customized ray external storage namespace * update GCS HA document remove duplicated constants Update docs/guidance/gcs-ha.md Co-Authored-By: Dmitri Gekhtman <62982571+DmitriGekhtman@users.noreply.github.com> * try not use UID as storage namespace Co-authored-by: Dmitri Gekhtman <62982571+DmitriGekhtman@users.noreply.github.com>

wilsonwang371 force-pushed the wilson/ray2.0_gcs_ha branch 6 times, most recently from 92a57ff to 0444021 Compare June 22, 2022 23:04

wilsonwang371 changed the title ~~initial support for external redis and gcs ha~~ [WIP]initial support for external redis and gcs ha Jun 23, 2022

wilsonwang371 force-pushed the wilson/ray2.0_gcs_ha branch 20 times, most recently from 7d4436c to a4119e7 Compare June 23, 2022 23:48

DmitriGekhtman reviewed Jul 19, 2022

View reviewed changes

docs/guidance/gcs-ha.md Outdated Show resolved Hide resolved

DmitriGekhtman reviewed Jul 19, 2022

View reviewed changes

ray-operator/controllers/ray/raycluster_controller.go Show resolved Hide resolved

DmitriGekhtman reviewed Jul 19, 2022

View reviewed changes

docs/guidance/gcs-ha.md Show resolved Hide resolved

DmitriGekhtman reviewed Jul 19, 2022

View reviewed changes

DmitriGekhtman approved these changes Jul 19, 2022

View reviewed changes

DmitriGekhtman mentioned this pull request Jul 19, 2022

Add Ray address env. #388

Merged

4 tasks

Jeffwan approved these changes Jul 19, 2022

View reviewed changes

wilsonwang371 commented Jul 19, 2022

View reviewed changes

ray-operator/controllers/ray/common/constant.go Outdated Show resolved Hide resolved

DmitriGekhtman mentioned this pull request Jul 20, 2022

[autoscaler][kuberay] Disable autoscaler health check and drain functionality ray-project/ray#26764

Merged

6 tasks

update GCS HA document

44c8184

remove duplicated constants Update docs/guidance/gcs-ha.md Co-Authored-By: Dmitri Gekhtman <62982571+DmitriGekhtman@users.noreply.github.com>

wilsonwang371 force-pushed the wilson/ray2.0_gcs_ha branch 3 times, most recently from c12e66a to 7e9c82c Compare July 20, 2022 18:40

try not use UID as storage namespace

6aeb740

wilsonwang371 force-pushed the wilson/ray2.0_gcs_ha branch from 7e9c82c to 6aeb740 Compare July 20, 2022 19:09

Jeffwan merged commit 2e0e9cd into ray-project:master Jul 20, 2022

wilsonwang371 mentioned this pull request Jul 24, 2022

enable ray external storage namespace #406

Merged

4 tasks

wilsonwang371 mentioned this pull request Jul 27, 2022

improve compatibility test stability #418

Merged

4 tasks

wilsonwang371 deleted the wilson/ray2.0_gcs_ha branch August 18, 2022 04:35

davidxia mentioned this pull request Jan 24, 2023

refactor: use equivalent, shorter Builder.Owns() method #881

Merged

4 tasks

jeevb mentioned this pull request Mar 2, 2023

[Bug][Operator][Leader election?] Operator failure and restart, logs attached #601

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for external Redis and GCS HA #294

Initial support for external Redis and GCS HA #294

wilsonwang371 commented Jun 8, 2022 •

edited

Loading

wilsonwang371 commented Jun 8, 2022

Jeffwan commented Jun 8, 2022 •

edited

Loading

wilsonwang371 commented Jun 9, 2022

DmitriGekhtman commented Jul 18, 2022

DmitriGekhtman Jul 19, 2022

wilsonwang371 Jul 19, 2022

DmitriGekhtman left a comment

DmitriGekhtman commented Jul 19, 2022 •

edited

Loading

DmitriGekhtman commented Jul 20, 2022

brucez-anyscale commented Jul 20, 2022

Jeffwan commented Jul 20, 2022

wilsonwang371 commented Jul 20, 2022

brucez-anyscale commented Jul 20, 2022

wilsonwang371 commented Jul 24, 2022

Initial support for external Redis and GCS HA #294

Initial support for external Redis and GCS HA #294

Conversation

wilsonwang371 commented Jun 8, 2022 • edited Loading

Why are these changes needed?

Changes

Enable GCS HA

Readiness Probe & Liveness Probe

Related issue number

Checks

wilsonwang371 commented Jun 8, 2022

Jeffwan commented Jun 8, 2022 • edited Loading

wilsonwang371 commented Jun 9, 2022

DmitriGekhtman commented Jul 18, 2022

DmitriGekhtman Jul 19, 2022

Choose a reason for hiding this comment

wilsonwang371 Jul 19, 2022

Choose a reason for hiding this comment

DmitriGekhtman left a comment

Choose a reason for hiding this comment

DmitriGekhtman commented Jul 19, 2022 • edited Loading

DmitriGekhtman commented Jul 20, 2022

brucez-anyscale commented Jul 20, 2022

Jeffwan commented Jul 20, 2022

wilsonwang371 commented Jul 20, 2022

brucez-anyscale commented Jul 20, 2022

wilsonwang371 commented Jul 24, 2022

wilsonwang371 commented Jun 8, 2022 •

edited

Loading

Jeffwan commented Jun 8, 2022 •

edited

Loading

DmitriGekhtman commented Jul 19, 2022 •

edited

Loading