Added role, rolebinding, serviceaccount, and environment variables. #233

aaronfern · 2021-09-20T15:29:37Z

Added role, rolebinding, serviceaccount, and environment variables to allow etcd-backup-restore sidecar to become kubernetes aware

How to categorize this PR?

/area backup
/area disaster-recovery
/area high-availability
/kind enhancement

What this PR does / why we need it:
As part of making the backup-restore sidecar kubernetes aware, this PR does the following

Allows druid to now create serviceaccount, role, and rolebinding and associates them to the etcd pods, to authenticate it to the API server.
Passes the pod name and current namespace to the backup-restore pod as environment variables
Removes logic for cronjob as it is to be replaced with trigger based compaction jobs instead

Which issue(s) this PR fixes:
Partially fixes gardener/etcd-backup-restore#333

Special notes for your reviewer:

Release note:

`druid` will now also reconcile a `serviceaccount`, a `role`, and a `rolebinding` as part of it's `etcd` reconcile flow and associate it with the etcd pod

timuthy · 2021-09-29T13:15:20Z

/assign

timuthy

Thanks for the PR. I left a few comments and questions.

charts/etcd/templates/etcd-role.yaml

charts/etcd/templates/etcd-rolebinding.yaml

charts/etcd/templates/etcd-serviceaccount.yaml

controllers/controller_ref_manager.go

shreyas-s-rao

@aaronfern Thanks for the PR. This PR will now need to be modified to remove cronjob, after out-of-band discussions to introduce trigger-based compaction jobs. Also, please remove the claim logic for the three resources you've introduced, as this is not a necessity, as mentioned here. Thanks.

timuthy

Thanks for addressing the feedback @aaronfern. After having a second look, I added some additional comments.

charts/etcd/templates/etcd-role.yaml

charts/etcd/templates/etcd-statefulset.yaml

config/crd/bases/druid.gardener.cloud_etcds.yaml

controllers/etcd_controller.go

timuthy · 2021-10-11T14:14:58Z

controllers/etcd_controller.go

-
-	if !backupCompactionScheduleFound {
-		if err == nil {
-			err = r.Delete(ctx, &cronJob, client.PropagationPolicy(metav1.DeletePropagationForeground))


I don't think that we can simply remove the deletion of cron jobs because people might have already enabled that feature, so these objects will remain in the cluster after switching to Job in #235.
In general I have to admit, I don't like that this PR tries to include both: Adding new resource for Backup-Restore and removing the CronJob behavior. Why?

It makes it depended on Enabled jobs instead of cronjobs to take care of running compaction. #235.

It breaks the main branch after merging.

It makes reviews harder to understand.

It makes the change history complex.
Not sure why that decision was made Added role, rolebinding, serviceaccount, and environment variables. #233 (review) (/cc @shreyas-s-rao).

@timuthy Unfortunately since we saw the cost impact of cron-schedule-based compaction only after making the release v0.6.0, we came up with the on-demand compaction job approach and decided to release it as a patch on v0.6.0 rather than a new release v0.7.0, to avoid making v0.6.0 a "dead" release (because v0.6.0, though technically usable, is not practically a viable solution).

I agree, we also need to include code to remove any possible cronjob resources in the cluster created previously, which should be ideally handled by #235 .

Agreed that it would make more sense that the switch from cronjob to job happens fully in #235 , while this PR only takes care of adding the RBAC required for backup sidecar's lease updation feature.

@abdasgupta WDYT? Can you add the cronjob removal code and removal of BackupCompactionSchedule in your PR with a breaking change release note?

Yes, I already took care of cron job related codes in my PR. Would add a release note. Thanks.

timuthy · 2021-10-11T14:15:35Z

api/v1alpha1/etcd_types.go

@@ -141,9 +141,6 @@ type BackupSpec struct {
 	// EnableProfiling defines if profiling should be enabled for the etcd-backup-restore-sidecar
 	// +optional
 	EnableProfiling *bool `json:"enableProfiling,omitempty"`
-	// BackupCompactionSchedule defines the cron standard for compacting the snapstore
-	// +optional
-	BackupCompactionSchedule *string `json:"backupCompactionSchedule,omitempty"`


We should have a breaking change release note for this.

Please undo this deletion. See previous comment about separation of duties between this PR and #235

shreyas-s-rao

@aaronfern Please remove the cronjob-removal related changes from your PR, since @abdasgupta will be including those changes in #235 with a breaking change release note. Let's keep this PR only for adding RBAC for backup sidecar to renew snapshot and member leases.

Please also add the relevant tests for sa, role, rb, like ensuring that the sa has read/write permissions on leases but not on other objects, and also ensuring that if the role/rb are not deployed, then sa cannot access leases as well.

shreyas-s-rao · 2021-10-22T08:04:14Z

charts/etcd/templates/etcd-compaction-cronjob.yaml

@@ -1,241 +0,0 @@
-apiVersion: batch/v1beta1


This file should ideally be removed in #235. Please undo this deletion here, since we don't want to mix the RBAC for backup sidecar along with cronjob removal. Those are two separate changes.
cc @abdasgupta

shreyas-s-rao · 2021-10-22T08:04:35Z

api/v1alpha1/zz_generated.deepcopy.go

@@ -87,11 +87,6 @@ func (in *BackupSpec) DeepCopyInto(out *BackupSpec) {
 		*out = new(bool)
 		**out = **in
 	}
-	if in.BackupCompactionSchedule != nil {


See previous comment.

shreyas-s-rao · 2021-10-22T08:05:01Z

api/v1alpha1/etcd_types.go

@@ -141,9 +141,6 @@ type BackupSpec struct {
 	// EnableProfiling defines if profiling should be enabled for the etcd-backup-restore-sidecar
 	// +optional
 	EnableProfiling *bool `json:"enableProfiling,omitempty"`
-	// BackupCompactionSchedule defines the cron standard for compacting the snapstore
-	// +optional
-	BackupCompactionSchedule *string `json:"backupCompactionSchedule,omitempty"`


Please undo this deletion. See previous comment about separation of duties between this PR and #235

shreyas-s-rao · 2021-10-22T08:11:02Z

config/crd/bases/druid.gardener.cloud_etcds.yaml

@@ -46,12 +46,6 @@ spec:
              backup:
                description: BackupSpec defines parametes associated with the full and delta snapshots of etcd
                properties:
-                  backupCompactionSchedule:


See previous comment about separation of tasks between this PR and #235

shreyas-s-rao · 2021-10-22T08:13:57Z

controllers/controller_ref_manager.go

@@ -31,8 +31,8 @@ import (
 	"k8s.io/client-go/tools/cache"

 	appsv1 "k8s.io/api/apps/v1"
-	batchv1 "k8s.io/api/batch/v1beta1"


Please see my previous comment about separation of tasks between this PR and #235. Also applies to the other cronjob related changes in this file as well.

shreyas-s-rao · 2021-10-22T08:14:49Z

controllers/controllers_suite_test.go

@@ -104,7 +104,7 @@ var _ = BeforeSuite(func(done Done) {
 	Expect(err).NotTo(HaveOccurred())

 	Expect(err).NotTo(HaveOccurred())
-	er, err := NewEtcdReconcilerWithImageVector(mgr, false)


See my previous comment about separation of tasks between this PR and #235

shreyas-s-rao · 2021-10-22T08:15:53Z

controllers/etcd_controller.go

@@ -791,142 +797,129 @@ func (r *EtcdReconciler) getStatefulSetFromEtcd(etcd *druidv1alpha1.Etcd, values
 	return decoded, nil
 }

-func (r *EtcdReconciler) reconcileCronJob(ctx context.Context, logger logr.Logger, etcd *druidv1alpha1.Etcd, values map[string]interface{}) (*batchv1.CronJob, error) {


See previous comment about separation of tasks between this PR and #235. Also applies to other cronjob-related changes in this file.

shreyas-s-rao · 2021-10-22T08:21:43Z

/needs rebase

… allow etcd-backup-restore sidecar to become kubernetes aware

timuthy

Just one nit, otherwise it looks good.

timuthy · 2021-10-26T13:09:55Z

controllers/etcd_controller_test.go

@@ -777,8 +804,90 @@ var _ = Describe("Druid", func() {
 		})
 	})

+	Describe("with etcd resource", func() {


Is it necessary to create a separate test case for the additional resources? I would suggest to add the validation to existing tests, e.g. DescribeTable("when etcd resource is created with backupCompactionSchedule field". This will save some testing time. WDYT?

Yes, that makes sense
Updated to add role validations to existing tests

timuthy

/lgtm

Reviewer is absent

aaronfern requested a review from a team as a code owner September 20, 2021 15:29

aaronfern force-pushed the k8s-aware branch from 87e5df5 to 60e57fe Compare September 22, 2021 12:57

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 22, 2021

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 22, 2021

gardener-robot assigned timuthy Sep 29, 2021

timuthy suggested changes Sep 29, 2021

View reviewed changes

gardener-robot added the needs/changes Needs (more) changes label Sep 29, 2021

shreyas-s-rao requested changes Sep 29, 2021

View reviewed changes

aaronfern force-pushed the k8s-aware branch from 60e57fe to 84f93b6 Compare October 6, 2021 05:40

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 6, 2021

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 6, 2021

abdasgupta changed the title ~~Added role, rolebinding, serviceaccount, and environment variables to…~~ Added role, rolebinding, serviceaccount, and environment variables. Oct 6, 2021

aaronfern mentioned this pull request Oct 7, 2021

Enabled jobs instead of cronjobs to take care of running compaction. #235

Merged

timuthy suggested changes Oct 11, 2021

View reviewed changes

aaronfern force-pushed the k8s-aware branch from 84f93b6 to 10c344f Compare October 12, 2021 15:26

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 12, 2021

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 12, 2021

shreyas-s-rao previously requested changes Oct 22, 2021

View reviewed changes

gardener-robot added needs/rebase Needs git rebase size/l Size of pull request is large (see gardener-robot robot/bots/size.py) and removed size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) labels Oct 22, 2021

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 22, 2021

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 22, 2021

gardener-robot added size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) and removed size/l Size of pull request is large (see gardener-robot robot/bots/size.py) labels Oct 22, 2021

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Oct 22, 2021

aaronfern added 3 commits October 22, 2021 19:43

Added role, rolebinding, serviceaccount, and environment variables to…

22b4d6f

… allow etcd-backup-restore sidecar to become kubernetes aware

Reverted cronjob code removal

25ee6d3

Added test to verify permissions granted by role

f10d61e

aaronfern force-pushed the k8s-aware branch from 9e72bba to f10d61e Compare October 22, 2021 14:38

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 22, 2021

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 22, 2021

timuthy reviewed Oct 26, 2021

View reviewed changes

Merged role test case with existing cronjob test cases

4b3028a

gardener-robot added size/l Size of pull request is large (see gardener-robot robot/bots/size.py) and removed size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) labels Oct 26, 2021

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Oct 26, 2021

timuthy approved these changes Nov 5, 2021

View reviewed changes

gardener-robot added reviewed/lgtm Has approval for merging and removed needs/changes Needs (more) changes needs/rebase Needs git rebase needs/review Needs review needs/second-opinion Needs second review by someone else labels Nov 5, 2021

timuthy merged commit 222a1cc into gardener:master Nov 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added role, rolebinding, serviceaccount, and environment variables. #233

Added role, rolebinding, serviceaccount, and environment variables. #233

aaronfern commented Sep 20, 2021 •

edited by shreyas-s-rao

Loading

timuthy commented Sep 29, 2021

timuthy left a comment

shreyas-s-rao left a comment

timuthy left a comment

timuthy Oct 11, 2021

shreyas-s-rao Oct 13, 2021

abdasgupta Oct 20, 2021

timuthy Oct 11, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao left a comment

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao Oct 22, 2021

shreyas-s-rao commented Oct 22, 2021

timuthy left a comment

timuthy Oct 26, 2021

aaronfern Oct 26, 2021

timuthy left a comment

Added role, rolebinding, serviceaccount, and environment variables. #233

Added role, rolebinding, serviceaccount, and environment variables. #233

Conversation

aaronfern commented Sep 20, 2021 • edited by shreyas-s-rao Loading

timuthy commented Sep 29, 2021

timuthy left a comment

Choose a reason for hiding this comment

shreyas-s-rao left a comment

Choose a reason for hiding this comment

timuthy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shreyas-s-rao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shreyas-s-rao commented Oct 22, 2021

timuthy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timuthy left a comment

Choose a reason for hiding this comment

aaronfern commented Sep 20, 2021 •

edited by shreyas-s-rao

Loading