Fix issue with operator OOM restart #946

wilsonwang371 · 2023-03-07T05:22:05Z

Why are these changes needed?

This is the patch to fix #601 issue.

In our kuberay controller, we are watching all the events so that we can find out if there is any failed health probe event. In case we see this event happening, we will try to create a new pod to replace the failed one currently. Ideally, we should not receive a large amount of these events. However, we are not correctly filtering out these messages. When a lot of unrelated events have arrived, we will end up printing too many log messages and make the controller run slow.

In this patch, we use filters to block the event messages that we do not care about as early as possible.

Related issue number

#601

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Jeffwan · 2023-03-07T19:38:27Z

For bug fixes, we normally add some tests to compare before & after. Could you add some tests?

Jeffwan

overall looks good to me

ray-operator/controllers/ray/raycluster_controller.go

kevin85421

Thanks for the contribution! I will check out this branch and test it by myself next week.

ray-operator/Makefile

kevin85421 · 2023-03-12T06:44:50Z

ray-operator/controllers/ray/raycluster_controller.go

+			"event", event)
+		return ctrl.Result{}, nil
+	} else if len(pods.Items) > 1 {
+		// This happens when we use fake client


Why do we need to have two Pods with the same name and same namespace in test mode?

this happens during test which we do not have the server side filtering available.

Would you mind explaining it more? In my understanding, it is impossible to have two Pods with the same name in an actual situation? Why do we need to test this case? Do I miss anything? Thanks!

This is happening during test in which we are using fakeclient which does not support these different kinds of filters applied in the ListOption. If we don't have this, the test will fail which is not because of a real failure.

ray-operator/controllers/ray/raycluster_controller.go

ray-operator/controllers/ray/raycluster_controller_fake_test.go

kevin85421 · 2023-03-12T07:14:19Z

ray-operator/controllers/ray/raycluster_controller_fake_test.go

+	testPodEventName := fmt.Sprintf("%s.15f0c0c5c5c5c5c5", testPodName)
+
+	// add a pod in a different namespace
+	newPods := []runtime.Object{


The specifications of the Pods are very similar (some parts are the same.). Can we use variables instead of expanding all of them?

This is a test case. In this test case, we have common.RayNodeLabelKey: "yes", specified. I know some parts are the same but putting them in a separate place might not be good for test cases.

Let me explain it more. My meaning is that:

spec := corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } &corev1.Pod { ... Spec: spec (Maybe need to use Deep copy) } &corev1.Pod { ... Spec: spec (Maybe need to use Deep copy) }

instead of

&corev1.Pod { ... Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } } &corev1.Pod { ... Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } }

This is not a single location that we need to make this change. I'd suggest we make another MR to change all of these.

There are a lot of technical debts in KubeRay now. It's good for us to reduce the system's complexity in every PR : )

ray-operator/controllers/ray/raycluster_controller_fake_test.go

kevin85421 · 2023-03-16T18:31:32Z

ray-operator/controllers/ray/raycluster_controller_fake_test.go

+	testPodEventName := fmt.Sprintf("%s.15f0c0c5c5c5c5c5", testPodName)
+
+	// add a pod in a different namespace
+	newPods := []runtime.Object{


Let me explain it more. My meaning is that:

spec := corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } &corev1.Pod { ... Spec: spec (Maybe need to use Deep copy) } &corev1.Pod { ... Spec: spec (Maybe need to use Deep copy) }

instead of

&corev1.Pod { ... Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } } &corev1.Pod { ... Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "ray-worker", Image: "rayproject/ray:2.2.0", Command: []string{"echo"}, Args: []string{"Hello Ray"}, }, }, } }

kevin85421 · 2023-03-16T18:36:18Z

ray-operator/controllers/ray/raycluster_controller.go

+			"event", event)
+		return ctrl.Result{}, nil
+	} else if len(pods.Items) > 1 {
+		// This happens when we use fake client


Would you mind explaining it more? In my understanding, it is impossible to have two Pods with the same name in an actual situation? Why do we need to test this case? Do I miss anything? Thanks!

ray-operator/controllers/ray/raycluster_controller.go

ray-operator/controllers/ray/raycluster_controller_fake_test.go

Jeffwan · 2023-03-22T03:06:11Z

@wilsonwang371 Please help address @kevin85421's comment.

kevin85421 · 2023-03-27T21:20:22Z

Hi @wilsonwang371, would you mind rebasing with the master's branch? Because of the tight deadline for the release v0.5.0, @architkulkarni and I will merge this PR today and file a new PR to fix/refactor if we find any bugs in this PR. Thanks!

architkulkarni

Overall looks good to me

architkulkarni · 2023-03-27T22:01:27Z

ray-operator/controllers/ray/raycluster_controller.go

 		return ctrl.Result{}, nil
 	}

 	_ = r.Log.WithValues("event", request.NamespacedName)
-	r.Log.Info("reconcile RayCluster Event", "event name", request.Name)


Why remove this log? Is it no longer useful?

architkulkarni · 2023-03-27T22:20:15Z

ray-operator/controllers/ray/raycluster_controller_fake_test.go

+		Log:      ctrl.Log.WithName("controllers").WithName("RayCluster"),
+	}
+
+	// add event for reconcile


Does it make sense to also add an event which should get filtered out and not trigger a reconcile?

architkulkarni · 2023-03-27T22:21:05Z

ray-operator/controllers/ray/raycluster_controller.go

+				CreateFunc: func(e event.CreateEvent) bool {
+					if eventObj, ok := e.Object.(*corev1.Event); ok {
+						if eventObj.InvolvedObject.Kind != "Pod" || eventObj.Type != "Warning" ||
+							eventObj.Reason != "Unhealthy" || !strings.Contains(eventObj.Message, "Readiness probe failed") {


Is there a more programmatic way to get the cause of the unhealthiness, or is scanning the message string standard practice? Also, for my education, why do we only care about failures in the readiness probe and not other pod unhealthy events? Maybe @kevin85421 you can also respond if you know the answer

readiness probe failure will trigger liveness probe failure. readiness probe will not automatically trigger a pod rebuild but liveness probe failure will.

architkulkarni · 2023-03-27T22:23:33Z

@wilsonwang371 Do you mind adding a bit more detail to the PR description?

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: Wilson Wang <3913185+wilsonwang371@users.noreply.github.com>

wilsonwang371 · 2023-03-27T23:52:44Z

Sorry, I am a little bit busy these days. I just took some time to rebase and update the comments. Let's wait and see how's @jeevb reply on this latest patch.

kevin85421 · 2023-03-28T06:38:54Z

ray-operator/controllers/ray/raycluster_controller.go

 )

 // NewReconciler returns a new reconcile.Reconciler
 func NewReconciler(mgr manager.Manager) *RayClusterReconciler {
+	if err := mgr.GetFieldIndexer().IndexField(context.Background(), &corev1.Pod{}, podUIDIndexField, func(rawObj client.Object) []string {
+		pod := rawObj.(*corev1.Pod)


The following change may fix #601 (comment). cc @jeevb

pod := rawObj.(*corev1.Pod) owner := metav1.GetControllerOf(pod) if owner == nil || owner.APIVersion != rayiov1alpha1.GroupVersion.String() || owner.Kind != "RayCluster" { return nil } return []string{string(pod.UID)}

^

Above is wrong. This is used to enable efficient lookup operations for the local cache. The informer will still fetch cluster status from Kubernetes APIServer. That is, events which are unrelated to Ray Pods will finally be fetched by the informer. Sorry for the misleading.

kevin85421

If you do not have time to update & test this PR tomorrow, I can merge this PR and open a new patch to fix the issues.

kevin85421 · 2023-03-28T06:40:34Z

ray-operator/controllers/ray/raycluster_controller.go

+	options := []client.ListOption{
+		client.MatchingFields(map[string]string{podUIDIndexField: string(event.InvolvedObject.UID)}),
+		client.InNamespace(event.InvolvedObject.Namespace),
+		client.MatchingLabels(map[string]string{common.RayNodeLabelKey: "yes"}),


Remove all common.RayNodeLabelKey

I found out that I misunderstood some things. Initially, I thought that we should implement all filter logic in the resource event handler (i.e. CreateFunc) and return true only when an event is issued by a Ray Pod. With this approach, we can make the function eventReconcile only handle event processing and not worry about event filtering. Therefore, the label common.RayNodeLabelKey is not necessary.

However, I have since realized that the owners of the unhealthy events are Pods, which may include Ray Pods and other Pods. Therefore, we cannot use GetControllerOf to get the owner and determine whether the event belongs to a Ray Pod or not by checking whether the owner's kind is "RayCluster" or not.

Hence, the implementation will be more complex. If we want to distinguish whether an event belongs to a Ray Pod or not, we need to

Get the UID of the event's InvolvedObject.

Use the UID to look up Pods from the local cache.

Use GetControllerOf(pod) to get the owner, and check whether the kind is RayCluster or not.

It will make the function eventReconcile easier to understand, but we need to have more I/O (read local cache) which may cause performance overhead.

In summary, we should use common.RayNodeLabelKey to distinguish Ray Pods in eventReconcile until we understand the cost of the look-up operation.

wilsonwang371 · 2023-03-28T06:45:13Z

If you do not have time to update & test this PR tomorrow, I can merge this PR and open a new patch to fix the issues.

If you have time to help with this, that will be good.

Jeffwan · 2023-03-28T17:03:01Z

@kevin85421 @architkulkarni If there're minor changes to be applied, I think we can check in the code and create separate PRs for improvement. If there're some minor changes you can directly suggest on the PR, feel free to use https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/incorporating-feedback-in-your-pull-request. That could reduce response time.

kevin85421 · 2023-03-28T17:29:08Z

Merge this PR, and will file a separate PR to fix and improve some minor issues.

Fix issue with operator OOM restart

wilsonwang371 requested review from DmitriGekhtman, Jeffwan and scarlet25151 March 7, 2023 05:22

wilsonwang371 force-pushed the wilson/fix-601 branch from 251edd7 to 361b22d Compare March 7, 2023 05:34

wilsonwang371 changed the title ~~[DO NOT MERGE]fix issue with operator OOM restart~~ [WIP]fix issue with operator OOM restart Mar 7, 2023

wilsonwang371 linked an issue Mar 7, 2023 that may be closed by this pull request

[Bug][Operator][Leader election?] Operator failure and restart, logs attached #601

Closed

2 tasks

kevin85421 self-requested a review March 7, 2023 23:16

wilsonwang371 changed the title ~~[WIP]fix issue with operator OOM restart~~ Fix issue with operator OOM restart Mar 8, 2023

wilsonwang371 self-assigned this Mar 8, 2023

wilsonwang371 added this to the v0.5.0 release milestone Mar 8, 2023

Jeffwan reviewed Mar 9, 2023

View reviewed changes

ray-operator/controllers/ray/raycluster_controller.go Outdated Show resolved Hide resolved

ray-operator/controllers/ray/raycluster_controller.go Outdated Show resolved Hide resolved

kevin85421 reviewed Mar 12, 2023

View reviewed changes

wilsonwang371 force-pushed the wilson/fix-601 branch 4 times, most recently from be253eb to 26c5593 Compare March 16, 2023 05:35

kevin85421 reviewed Mar 16, 2023

View reviewed changes

wilsonwang371 force-pushed the wilson/fix-601 branch 4 times, most recently from 32776b4 to 820a135 Compare March 21, 2023 21:03

architkulkarni self-requested a review March 27, 2023 18:28

architkulkarni approved these changes Mar 27, 2023

View reviewed changes

wilsonwang371 added 3 commits March 27, 2023 15:34

fix issue with operator oom restart

57efb07

add error checking

7cf6328

add unittest for new event reconcile logic

d46ff32

wilsonwang371 and others added 6 commits March 27, 2023 15:34

add err checking

1ce2b9c

update code after code comments

866b5b5

debug message

19a8625

Update ray-operator/controllers/ray/raycluster_controller.go

2aac332

Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: Wilson Wang <3913185+wilsonwang371@users.noreply.github.com>

add manager indexfield call

cea23be

update code

61173bf

wilsonwang371 force-pushed the wilson/fix-601 branch from b6eb222 to 61173bf Compare March 27, 2023 22:34

kevin85421 reviewed Mar 28, 2023

View reviewed changes

Jeffwan approved these changes Mar 28, 2023

View reviewed changes

kevin85421 merged commit 3c53af6 into ray-project:master Mar 28, 2023

kevin85421 mentioned this pull request Mar 29, 2023

[Feature] Optimize indexer #996

Closed

4 tasks

This was referenced Apr 5, 2023

[Bug] Operator watch resource needs to be optimized #403

Closed

[Bug] GCS HA pod event handling emits logs about irrelevant pods #785

Closed

lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023

Fix issue with operator OOM restart (ray-project#946)

404571f

Fix issue with operator OOM restart

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue with operator OOM restart #946

Fix issue with operator OOM restart #946

wilsonwang371 commented Mar 7, 2023 •

edited

Loading

Jeffwan commented Mar 7, 2023

Jeffwan left a comment

kevin85421 left a comment

kevin85421 Mar 12, 2023

wilsonwang371 Mar 16, 2023

kevin85421 Mar 16, 2023

wilsonwang371 Mar 22, 2023

kevin85421 Mar 12, 2023

wilsonwang371 Mar 16, 2023

kevin85421 Mar 16, 2023

wilsonwang371 Mar 21, 2023

kevin85421 Mar 21, 2023

kevin85421 Mar 16, 2023

kevin85421 Mar 16, 2023

Jeffwan commented Mar 22, 2023

kevin85421 commented Mar 27, 2023

architkulkarni left a comment

architkulkarni Mar 27, 2023

architkulkarni Mar 27, 2023

architkulkarni Mar 27, 2023

wilsonwang371 Mar 28, 2023

architkulkarni commented Mar 27, 2023

wilsonwang371 commented Mar 27, 2023

kevin85421 Mar 28, 2023

kevin85421 Mar 29, 2023

kevin85421 left a comment

kevin85421 Mar 28, 2023

wilsonwang371 Mar 28, 2023

kevin85421 Mar 29, 2023 •

edited

Loading

wilsonwang371 commented Mar 28, 2023

Jeffwan commented Mar 28, 2023 •

edited

Loading

kevin85421 commented Mar 28, 2023

Fix issue with operator OOM restart #946

Fix issue with operator OOM restart #946

Conversation

wilsonwang371 commented Mar 7, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

Jeffwan commented Mar 7, 2023

Jeffwan left a comment

Choose a reason for hiding this comment

kevin85421 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jeffwan commented Mar 22, 2023

kevin85421 commented Mar 27, 2023

architkulkarni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

architkulkarni commented Mar 27, 2023

wilsonwang371 commented Mar 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 Mar 29, 2023 • edited Loading

Choose a reason for hiding this comment

wilsonwang371 commented Mar 28, 2023

Jeffwan commented Mar 28, 2023 • edited Loading

kevin85421 commented Mar 28, 2023

wilsonwang371 commented Mar 7, 2023 •

edited

Loading

kevin85421 Mar 29, 2023 •

edited

Loading

Jeffwan commented Mar 28, 2023 •

edited

Loading