[GEP-7] Control Plane Migration: Etcd Backup Migration / Leader Election PoC

[stoyanr]

How to categorize this issue?

/area control-plane-migration
/kind enhancement
/priority 3

What would you like to be added:

We would like to implement a PoC for the etcd backup migration / leader election mechanism as currently proposed with #3066, in order to evaluate the feasibility of the proposed approach and try to address the remaining open issues. We don't aim yet for production quality code and won't be opening PRs to gardener components while still at the PoC stage.

Below is a rough outline of what we would like to implement as part of the PoC. Based on what we learned or ongoing discussions, we might change direction and implement things differently. In such cases, we will update this issue accordingly.

1. gardenlet restoration flow

• Creates one additional BE for source seed, changes the seed name to the dest seed in the existing BE
• Annotates the etcd resource with operation=restore
• Waits for the status of the etcd resource to indicate that the migration is finished
• Annotates the etcd resource with operation=reconcile
• Deletes the source seed BE

2. etcd-druid

• Reconciles an etcd resource with operation=restore and creates etcd-backup-restore migrate job
• Waits for the etcd-backup-restore migrate job to complete and updates the status of the etcd resource

3. BE controller

• Handles 2 BEs correctly to avoid name collisions in secrets

4. etcd-backup-restore support for migrate

• Writes header files in source and dest BBs for the migrate operation
• Waits for a snapshot newer than the header timestamp to be created (final snapshot) with a timeout (e.g. 10 min)
• Migrates all snapshots from source to dest BB (final snapshot first, rest after that)
• Updates header files that the migration was performed

5. etcd-backup-restore normal operation

• Checks header files before taking a snapshot
• If source detected
  • If final snapshot not taken (how to detect?), cuts access to main etcd (how?), takes final snapshot, and exits
  • Otherwise, just exits
• If destination detected
  • If a snapshot exists, restores it
  • Otherwise, waits with a timeout, then exits
• Otherwise, takes a normal snapshot as before

[stoyanr]

/assign @stoyanr @plkokanov @kris94
/cc @rfranzke @mandelsoft @amshuman-kr This is PoC / experimentation, if you would rather have us try things differently, please comment. Otherwise, we would in any case share and discuss the results with you in order to agree on a final approach.

[vlerenc]

@stoyanr I think I understand 2, 4, and 5, but can you elaborate a bit more on 1 and 3?

[stoyanr]

I think I understand 2, 4, and 5, but can you elaborate a bit more on 1 and 3?

1 is simply the changes to the overall restoration flow (executed by gardenlet) to bind everything together and actually perform a successful "restore" reconciliation, provided that all other changes are in place.
3 is needed to ensure that if the BE controller has to reconcile 2 different BEs and therefore create 2 etcd-backup secrets for the same shoot, they are created with different names; currently the name is hardcoded to etcd-backup.

I apologise for the lack of in-depth technical explanations, but this issue is only intended to be a starting point for us to work on the PoC. In the above list, we are likely still missing some things, and perhaps also have a few that are not really needed. We'll be figuring this out as we go.

[stoyanr]

Closing this as the PoC is considered finished and GEP-17 is now approved.