migrate leader election to lease API

[ricky1993]

Change-Id: I21fd5cdc1af59e456628cf15fc84b2d79db2eda0

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
Re-implement #80508, due to the suggestion at #80508 (comment)

If user uses endpoint lock and want to migrate to lease lock, he can switch all components(kcm, scheduler, etc) to "endpointlease" lock and then switch to lease lock safely. Note that the old endpoint lock will not be clean.
Which issue(s) this PR fixes:

Ref #80289

Special notes for your reviewer:
Implement two composite resource locks for migration between lease and endpoint and configmap.

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig scalability
/assign @wojtek-t
/cc @mikedanese @timothysc any suggestions?

[k8s-ci-robot]

Hi @ricky1993. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wojtek-t]

/ok-to-test

This looks reasonable to me.
@mikedanese @timothysc - thoughts?

[timothysc]

I'm still confused by your motivations for this change, and imo release notes are required:

If user uses endpoint lock and want to migrate to lease lock, he can switch all components(kcm, scheduler, etc) to "endpointlease" lock and then switch to lease lock safely. Note that the old endpoint lock will not be clean.

^ What is the user story driving this? You want to switch the locking on the fly, that seems like a really bad idea? What am I missing here?

/hold

[ricky1993]

^ What is the user story driving this? You want to switch the locking on the fly, that seems like a really bad idea? What am I missing here?

The ref issue.
"Given that both of these are watched by different components, this is generating a lot of unnecessary load.

We should migrate all leader-election to use Lease API (that was designed exactly for this case)."

[wojtek-t]

You want to switch the locking on the fly, that seems like a really bad idea? What am I missing here?

@timothysc - endpoints or configmaps are not the objects that we should be doing leader election against. We have an API that was designed exactly for this purpose, which is coordination.Lease.
We would like to use in our components what we recommend others.
In addition to that, using Endpoints object for leader election is bad from performance/scalability POV - Endpoints are watched by kube-proxy on every single node, which means we don't really want to use Endpoints object.

And we exactly don't want to change it randomly but in two phases:
(1) acquire old object lock, once acquired, acquire new object lock; only when both acquired we have a lock and can proceed
(2) (next release) since now master has to acquire the lock on the new object (lease) to be master, we can now delete the need to acquire the old object (endpoints or configmap)
And now we're using lease as it should be.

Is that more clear now?

[ricky1993]

/test pull-kubernetes-e2e-gce

[fedebongio]

/cc @liggitt

[timothysc]

We have an API that was designed exactly for this purpose, which is coordination.Lease.
We would like to use in our components what we recommend others.

lol, we argued for this years ago but briant grant said no way. I don't know how you slid this through, but works for me.

/hold cancel

[timothysc]

/assign @liggitt
^ I wasn't involved in this api creation so deferring to api-approver.

[ricky1993]

@wojtek-t @mikedanese PTAL. I will add some unittest for multilock at "staging/src/k8s.io/client-go/tools/leaderelection/leaderelection_test.go" soon.

[ricky1993]

/test pull-kubernetes-kubemark-e2e-gce-big

[wojtek-t]

I didn't carefully reviewed the test yet, but the non-test logic looks reasonable for me.
But before I will review the test, I would prefer someone else to also review this.

@mikedanese - can you please take a look?

[ricky1993]

friendly ping~ @mikedanese

[mikedanese]

I would drop RawRecord from the public resource lock API. I don't think you need it. Other than that, this is what I would expect.

[wojtek-t]

Just two minor comments - other than that lgtm

[wojtek-t]

LGTM - will let @mikedanese to make a final look

[mikedanese]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikedanese, ricky1993

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/client-go/tools/leaderelection/OWNERS [mikedanese]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment