Removed owner reference from etcd statefulsets #48

georgekuruvillak · 2020-04-23T06:14:33Z

What this PR does / why we need it:
This PR removes the dependance on ownerReference in etcd statefulset. It has been observerd that ownerReference in statefulset prevents VPA from recommending values for vertical scaling.
Which issue(s) this PR fixes:
Fixes #47
Special notes for your reviewer:

Release note:

Removed owner reference from etcd StatefulSet so that HVPA can recommend resource recommendations. VPA does not support StatefulSet having `ownerReferences` set to another top-level controller .

amshuman-kr

Overall looks good. Can we add the test cases to cover at least the following edge cases?

Reconciliation of new etcd resource deployment without any existing statefulsets.
Reconciliation of new etcd resource deployment with adoption of existing statefulset.
Reconciliation of existing etcd resource with statefulset with ownerReference.
Reconciliation of deletion of etcd resource with statefulset with ownerReference.
Reconciliation of deletion of etcd resource with statefulset without ownerReference but with owned-by annotations.
Reconciliation of deletion of etcd resource with statefulset without ownerReference and without owned-by annotations.

Also, can you please add the reasoning to the release notes?

controllers/etcd_controller.go

swapnilgm

Thanks you for PR. Please address the comments.

controllers/controller_ref_manager.go

controllers/etcd_controller.go

controllers/controller_ref_manager.go

swapnilgm

As discussed, i thought of refactoring controller-ref-manager.go as separate task. But since we are touching part of it, let's at-least refactor and reusae the code that is being touched here.

controllers/etcd_controller.go

charts/etcd/templates/etcd-bootstrap-configmap.yaml

charts/etcd/templates/etcd-service.yaml

controllers/controller_ref_manager.go

controllers/etcd_controller_test.go

controllers/suite_test.go

controllers/etcd_controller_test.go

controllers/controller_ref_manager.go

…elds

…tatefulsets

swapnilgm

LGTM. Thanx for the test. I have tested the shoot creation/update with Gardener as well. Looks fine.

@ggaurav10 Will you please test against VPA/HVPA? Post that we can merge the PR and release.

ggaurav10 · 2020-04-27T14:41:40Z

@georgekuruvillak and I have already tested working from VPA/HVPA side, and it worked as expected.

amshuman-kr

Thanks for the test cases! Suggested some changes.

Apart from the suggestions below, I see quite a bit of code copied with variations in the test cases. Would it make sense to clean them up? Table tests could help.

api/v1alpha1/etcd_types.go

controllers/controller_ref_manager.go

amshuman-kr · 2020-04-28T04:13:01Z

controllers/etcd_controller_test.go

+				time.Sleep(2 * time.Second)
+				err = c.Delete(context.TODO(), instance)
+				Expect(err).NotTo(HaveOccurred())
+				err = retryTillStatefulSetRemoved(c, &sts)


Please use Eventually instead.

amshuman-kr · 2020-04-28T04:13:05Z

controllers/etcd_controller_test.go

+				Expect(err).NotTo(HaveOccurred())
+				err = retryTillStatefulSetRemoved(c, &sts)
+				Expect(err).NotTo(HaveOccurred())
+				err = retryTillEtcdRemoved(c, instance)


Please use Eventually instead.

amshuman-kr · 2020-04-28T04:13:23Z

controllers/etcd_controller_test.go

 })

-func getStatefulset(c client.Client, instance *druidv1alpha1.Etcd, ss *appsv1.StatefulSet) {
+func retryTillPodsDeleted(c client.Client, etcd *druidv1alpha1.Etcd) (*corev1.Pod, error) {


Please use Eventually instead.

amshuman-kr · 2020-04-28T04:13:32Z

controllers/etcd_controller_test.go

+	return pod, err
+}
+
+func retryTillEtcdRemoved(c client.Client, etcd *druidv1alpha1.Etcd) error {


Please use Eventually instead.

controllers/etcd_controller_test.go

georgekuruvillak · 2020-04-28T13:59:33Z

@amshuman-kr I have changed the tests to table tests. PTAL

amshuman-kr

Thanks a lot for refactoring the tests. Could you please add a couple of test cases (one each for creation and deletion scenario) which are small variations on the existing cases. I think they would strengthen druid against some common use-cases.

amshuman-kr · 2020-04-28T14:17:01Z

controllers/etcd_controller_test.go

+			case WithOwnerReference:
+				ss = createStatefulsetWithOwnerReference(instance)
+				Expect(len(ss.OwnerReferences)).ShouldNot(BeZero())
+			default:


Should we have a case for pre-existing StatefulSet with annotations? I know this won't happen when we make the next release. But it would be good to cover the case which would happen for every reconciliation after that. WDYT?

Pondering upon how the test would be like, what would be the expected behavior of the test. That the annotations still persist after the reconcile?

controllers/etcd_controller_test.go

amshuman-kr · 2020-04-28T14:22:27Z

controllers/etcd_controller_test.go

+			Eventually(func() error { return statefulSetRemoved(c, &sts) }, timeout, pollingInterval).Should(BeNil())
+			Eventually(func() error { return etcdRemoved(c, instance) }, timeout, pollingInterval).Should(BeNil())
+		},
+		Entry("when  statefulset with ownerReference and without owner annotations, druid should adopt and delete statefulset", "foo61", WithOwnerReference, false),


Should we have a case of pre-existing StatefulSet with no ownerReference and no annotations in which case the it should not be removed?

Actually, if there is no owner reference/ owner annotations in the sts, the druid will adopt it if the selector match. This is because, in the pre druid scenario, we didnt have both and druid went ahead adopting it. Should we not continue the behavior? I have written the test for the same in the following line below.

But it won't adopt if the etcd being deleted, no? Then the StatefulSet would be left untouched. A negative case.

Can you please make the change for the negative case, I mentioned above? I did not mean the adoption+deletion case.

Based on offline conversation. We can take it forward to another PR since this requires reconcile call to be brought in into the test func.

amshuman-kr

Thanks a lot for the test cases! The one open case can be addressed in another PR.

Removed owner ref from etcd sts

bd043e4

georgekuruvillak requested a review from swapnilgm April 23, 2020 06:14

georgekuruvillak requested a review from a team as a code owner April 23, 2020 06:14

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 23, 2020

georgekuruvillak changed the title ~~Removed owner reference from etcd statefulsets~~ [Do-Not-Merge] Removed owner reference from etcd statefulsets Apr 23, 2020

Modifying test cases to reflect removal of ownerRef

cadbe23

gardener-robot-ci-3 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Apr 23, 2020

georgekuruvillak changed the title ~~[Do-Not-Merge] Removed owner reference from etcd statefulsets~~ Removed owner reference from etcd statefulsets Apr 23, 2020

amshuman-kr requested changes Apr 23, 2020

View reviewed changes

controllers/etcd_controller.go Show resolved Hide resolved

swapnilgm suggested changes Apr 23, 2020

View reviewed changes

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 23, 2020

georgekuruvillak requested a review from swapnilgm April 23, 2020 15:53

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from a4c7ba8 to e2f6419 Compare April 23, 2020 15:56

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 24, 2020

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from e2f6419 to 5ef66e1 Compare April 24, 2020 07:27

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 24, 2020

swapnilgm suggested changes Apr 24, 2020

View reviewed changes

swapnilgm reviewed Apr 24, 2020

View reviewed changes

controllers/controller_ref_manager.go Outdated Show resolved Hide resolved

georgekuruvillak force-pushed the remove_ownerref_from_sts branch 2 times, most recently from ce80ff5 to f64df55 Compare April 25, 2020 09:18

review suggestions

9a76a09

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from f64df55 to 9a76a09 Compare April 25, 2020 09:49

I308301 added 6 commits April 25, 2020 17:36

Added test to check pod restarts on reconcile of failed statefulset

48a4f01

Added test case to validate sts spec change on etcd resource update.

664f77e

Added test to check pod restarts on reconcile of failed statefulset

72cfa6f

Added test case to validate sts spec change on etcd resource update.

dca06b7

Added tests to verify sts reconciliation reflects etcd.Spec parent fi…

fe77b02

…elds

Added testcases for ownerRef to owner annotation conversion in etcd s…

8cede83

…tatefulsets

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from f0ed0aa to 8cede83 Compare April 26, 2020 16:36

georgekuruvillak requested review from swapnilgm and amshuman-kr April 26, 2020 16:36

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 27, 2020

swapnilgm approved these changes Apr 27, 2020

View reviewed changes

amshuman-kr requested changes Apr 28, 2020

View reviewed changes

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Apr 28, 2020

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from d07660e to b40cf63 Compare April 28, 2020 13:59

georgekuruvillak requested a review from amshuman-kr April 28, 2020 14:00

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from b40cf63 to 14d3f4d Compare April 28, 2020 14:05

amshuman-kr requested changes Apr 28, 2020

View reviewed changes

Changed tests to table tests.

28b3c0e

georgekuruvillak force-pushed the remove_ownerref_from_sts branch from 14d3f4d to 28b3c0e Compare April 28, 2020 15:35

amshuman-kr approved these changes Apr 28, 2020

View reviewed changes

georgekuruvillak merged commit 6838160 into gardener:master Apr 28, 2020

georgekuruvillak deleted the remove_ownerref_from_sts branch April 29, 2020 07:25

swapnilgm mentioned this pull request Apr 30, 2020

[BUG] HVPA not able to scale etcd. #47

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removed owner reference from etcd statefulsets #48

Removed owner reference from etcd statefulsets #48

georgekuruvillak commented Apr 23, 2020 •

edited by swapnilgm

Loading

amshuman-kr left a comment •

edited

Loading

swapnilgm left a comment

swapnilgm left a comment

swapnilgm left a comment •

edited

Loading

ggaurav10 commented Apr 27, 2020

amshuman-kr left a comment

amshuman-kr Apr 28, 2020

amshuman-kr Apr 28, 2020

amshuman-kr Apr 28, 2020

amshuman-kr Apr 28, 2020

georgekuruvillak commented Apr 28, 2020

amshuman-kr left a comment

amshuman-kr Apr 28, 2020

georgekuruvillak Apr 28, 2020

amshuman-kr Apr 28, 2020

georgekuruvillak Apr 28, 2020 •

edited

Loading

amshuman-kr Apr 28, 2020 •

edited

Loading

amshuman-kr Apr 28, 2020

amshuman-kr Apr 28, 2020

amshuman-kr left a comment

Removed owner reference from etcd statefulsets #48

Removed owner reference from etcd statefulsets #48

Conversation

georgekuruvillak commented Apr 23, 2020 • edited by swapnilgm Loading

amshuman-kr left a comment • edited Loading

Choose a reason for hiding this comment

swapnilgm left a comment

Choose a reason for hiding this comment

swapnilgm left a comment

Choose a reason for hiding this comment

swapnilgm left a comment • edited Loading

Choose a reason for hiding this comment

ggaurav10 commented Apr 27, 2020

amshuman-kr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgekuruvillak commented Apr 28, 2020

amshuman-kr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgekuruvillak Apr 28, 2020 • edited Loading

Choose a reason for hiding this comment

amshuman-kr Apr 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amshuman-kr left a comment

Choose a reason for hiding this comment

georgekuruvillak commented Apr 23, 2020 •

edited by swapnilgm

Loading

amshuman-kr left a comment •

edited

Loading

swapnilgm left a comment •

edited

Loading

georgekuruvillak Apr 28, 2020 •

edited

Loading

amshuman-kr Apr 28, 2020 •

edited

Loading