[KEP-5366] Graceful Leader Transition

[Jefftree]

• One-line PR description: Allow leader elected components to transition to a follower state without restarting the process

• Issue link: Graceful Leader Transition #5366

• Other comments: WIP POC: [DO-NOT-MERGE] POC of Graceful Leader Transition kubernetes#132078

/sig api-machinery
/assign @jpbetz

[Jefftree]

/cc @michaelasp

[k8s-ci-robot]

@Jefftree: GitHub didn't allow me to request PR reviews from the following users: michaelasp.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @michaelasp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jpbetz]

Two comments then LGTM for alpha.

[jpbetz]

@deads2k Any concerns with an off-by-default alpha for this? This gives us a good place to start listing out risks..

[deads2k]

@deads2k Any concerns with an off-by-default alpha for this? This gives us a good place to start listing out risks..

So high level feedback (I haven't done a detailed read yet).

1. Refactoring to allow graceful lease release followed by os.Exit as normal is a standard refactor that doesn't require an approved KEP, so we can do independent of this being merged.
2. Changes to avoid the os.Exit are notably riskier, do require a KEP, do require featuregating, and do require testing that demonstrates that we can catch and stop all go func, all reflectors, and clear all caches.

I will circle back for a deeper review hopefully today, maybe Monday, but I'm hoping we're on same page about the two different things. I suspect that my 1 is prereq to 2.

[jpbetz]

I'm hoping we're on same page about the two different things. I suspect that my 1 is prereq to 2.

This is how I've been thinking about it too. Stating this directly in the KEP would make sense to me. Even with 1 not requiring a KEP, stating it as a required prelim of the work that does require a KEP sounds like a good way to approach this.

[deads2k]

Maybe an old iteration of the doc has mingled with new, but the user stories indicate a desire to dynamically select which controllers are running in a process. The design and graduation, and PRR don't reflect this. Please reconcile the design one way or the other. My personal suggestion: don't dynamically change which controllers and running and simply get graceful release and potentially (if you really like it) avoiding process restart.

TBH, once we flush the caches, I'm not entirely clear on what benefit we get from re-using the process.

[deads2k]

Big picture, once we flush the caches on graceful lease release, what is the benefit of re-using the process instead of exiting?

[deads2k]

sig and PRR lgtm

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, Jefftree

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [deads2k]
• keps/sig-api-machinery/OWNERS [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment