Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Leader election to backup-restore and allow only backup-restore Leader to take and uploads the snapshots. #353

Merged
merged 12 commits into from
Dec 31, 2021
Merged

Add Leader election to backup-restore and allow only backup-restore Leader to take and uploads the snapshots. #353

merged 12 commits into from
Dec 31, 2021

Conversation

ishan16696
Copy link
Member

@ishan16696 ishan16696 commented Jul 2, 2021 •
edited
Loading

What this PR does / why we need it:
Added leader-election to the backup-restore sidecar.

Added support to bring the snapshotters (both full and incremental) and defragmentation go routines under the control of the leader-election.

To take out of sync full/delta snapshot and to get the latest snapshot metadata, delegate the incoming HTTP/HTTPS request from backup Follower to the backup Leader.

Which issue(s) this PR fixes:
Fixes #321 Part-1
Fixes #354

Special notes for your reviewer:

Release note:

Added the leader-election to the backup-restore. Added support to allow only backup-restore Leader to take/upload the snapshots and trigger the defragmentation.

@ishan16696 ishan16696 requested a review from a team as a code owner July 2, 2021 07:27
@gardener-robot gardener-robot added needs/review Needs review size/l Size of pull request is large (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels Jul 2, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 2, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 2, 2021
amshuman-kr
amshuman-kr suggested changes Jul 2, 2021
View reviewed changes
Copy link
Collaborator

@amshuman-kr amshuman-kr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the first-cut implementation @ishan16696. I have some comments and questions below.

example/00-backup-restore-server-config.yaml Outdated Show resolved Hide resolved
pkg/leaderelection/init.go Outdated Show resolved Hide resolved
pkg/leaderelection/init.go Outdated Show resolved Hide resolved
pkg/leaderelection/init.go Outdated Show resolved Hide resolved
pkg/leaderelection/leaderelection.go Outdated Show resolved Hide resolved
pkg/leaderelection/leaderelection.go Outdated Show resolved Hide resolved
pkg/server/backuprestoreserver.go Outdated Show resolved Hide resolved
pkg/server/backuprestoreserver.go Outdated Show resolved Hide resolved
pkg/server/types.go Outdated Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter.go Outdated Show resolved Hide resolved
@gardener-robot gardener-robot added the needs/changes Needs (more) changes label Jul 2, 2021
@gardener-robot
Copy link

@ishan16696 You have pull request review with status CHANGES_REQUESTED, please check

@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 14, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 14, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 14, 2021
@shreyas-s-rao shreyas-s-rao added this to the v0.14.0 milestone Jul 21, 2021
@gardener-robot gardener-robot added size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) and removed size/l Size of pull request is large (see gardener-robot robot/bots/size.py) labels Jul 25, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 25, 2021
@ishan16696 ishan16696 changed the title [WIP] [Feature]Add Leader election to backup-restore and allow only backup-restore Leader to take and uploads the snapshots. [Feature]Add Leader election to backup-restore and allow only backup-restore Leader to take and uploads the snapshots. Jul 26, 2021
@ishan16696 ishan16696 mentioned this pull request Jul 27, 2021
Closed
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 13, 2021
@gardener-robot-ci-1 gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 13, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 6, 2021
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 7, 2021
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 7, 2021
@ishan16696 ishan16696 mentioned this pull request Dec 10, 2021
Merged
@ishan16696
Added the LeaderElection for backup-restore.
dd24af4 
Added support to allow only backup-restore Leader to take/uploads the snapshots.
@ishan16696
Added support to allow only backup-restore Leader to trigger the defr…
d078ef9 
…agmentation on etcd leader as well as on etcd follower.
@ishan16696
To take out-of-sync full/delta snapshot delegate the incoming http re…
b08a8a0 
…quest from backup Follower to backup Leader.
@ishan16696
Added the LeaderElection support for etcd-event.
b71b64b 
Merged the runServerWithoutSnapshotter and runServerWithSnapshotter function calls.
@ishan16696
Updated the config yaml file with healthConfig.
406d798
@ishan16696
Addressed the review comments.
00d5709
@ishan16696
Incorporate the member lease renewal feature with multi-node etcd.
9ef6153
@ishan16696
replaced the etcd client with etcd Maintenance/Cluster client.
e772e7f
@ishan16696
Fixed a bug in RenewMemberLease pkg.
a429301
@ishan16696
Refactored some functions to add unit tests.
3e5f65d
@ishan16696
Added the unit tests.
f836e65
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 20, 2021
aaronfern
aaronfern reviewed Dec 22, 2021
View reviewed changes
Copy link
Contributor

@aaronfern aaronfern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one query from me @ishan16696
Thanks

pkg/server/backuprestoreserver.go Show resolved Hide resolved
abdasgupta
abdasgupta suggested changes Dec 22, 2021
View reviewed changes
Copy link
Contributor

@abdasgupta abdasgupta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an initial look. Overall, great effort. Thanks.

pkg/leaderelection/leaderelection.go Show resolved Hide resolved
pkg/leaderelection/leaderelection.go Show resolved Hide resolved
pkg/leaderelection/leaderelection.go Outdated Show resolved Hide resolved
pkg/leaderelection/types.go Outdated Show resolved Hide resolved
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 27, 2021
@abdasgupta
Copy link
Contributor

\lgtm

@abdasgupta abdasgupta merged commit b6adfb0 into gardener:master Dec 31, 2021
This was referenced Jan 17, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aaronfern aaronfern aaronfern left review comments

@stoyanr stoyanr stoyanr requested changes

@abdasgupta abdasgupta abdasgupta approved these changes

@amshuman-kr amshuman-kr Awaiting requested review from amshuman-kr

Assignees
No one assigned
Labels
needs/changes Needs (more) changes needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/rebase Needs git rebase needs/review Needs review needs/second-opinion Needs second review by someone else size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py)
Projects
None yet
Milestone
v0.14.0
10 participants
@ishan16696 @gardener-robot @abdasgupta @stoyanr @amshuman-kr @aaronfern @gardener-robot-ci-1 @shreyas-s-rao @gardener-robot-ci-2 @gardener-robot-ci-3