✨Add option to use Lease lock if coordination group is available #600

mszostok · 2019-09-11T17:56:34Z

Description

This PR adds new strategy for NewResourceLock method: Choose the Lease lock if lease.coordination.k8s.io is available otherwise, use ConfigMaps.
Add tests coverage for implemented logic using ginko & gomega
Remove outdated information about cert generation (cert generation was removed in this PR: ⚠️ split webhook server and manifest generation #300)

Notes to reviewer:

the NewResourceLock method is not easily testable. To do it you need to hack it via injected rest config. I did it in a similar way as in kubectl pkg.
e2e tested with k8s 1.15 (kind)
unit tests can be changed to table tests, see this: https://gist.github.com/mszostok/11278dbc9316d526cd8a2e09f063e50f
but IMO is not worth it.

Fixes: #460

needs to be sync with #444

k8s-ci-robot · 2019-09-11T17:56:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mszostok
To complete the pull request process, please assign pwittrock
You can assign the PR to them by writing /assign @pwittrock in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mszostok · 2019-09-11T19:05:05Z

I've checked the #444 and there is no merge conflict because the logic that I'm adding in this PR is directly in NewResourceLock func and for #444 it should be like a "blackbox"

one thing that needs to be changed are comments:

	// LeaderElectionID determines the name of the configmap that leader election
	// will use for holding the leader lock.

we should remove the information about the configmaps cause it's implementation details
so on my PR, I've changed that into

	// LeaderElectionID determines the name of the resource lock that leader election
	// will use for holding the leader lock.

If we agree on that then on #444 it should be adjusted too

/cc @DirectXMan12

pkg/leaderelection/doc.go

pkg/leaderelection/leader_election.go

- Choose the Lease lock if lease.coordination.k8s.io is available otherwise, use ConfigMaps. - Add tests coverage for implemented logic using ginko & gomega - Remove outdated information about cert generation

mszostok · 2019-09-12T13:10:16Z

I had to rebase after merging #591

DirectXMan12

Should this be configurable?

Should we try pass in the RESTMapper instead of adding yet another call to discovery?

mszostok · 2019-09-21T18:50:48Z

@DirectXMan12 thanks for comments, please check my answers

Should we try pass in the RESTMapper instead of adding yet another call to discovery?

tbh I didn’t take this into account because this is only one call to api-server and only at the beginning. Using client allows us to encapsulate new logic inside the NewResourceLock. Personally, for me, it’s not a case for optimization but here is draft PR with RESTMapper: https://github.com/mszostok/controller-runtime/pull/1/files

if you think that it's a better approach then I will rewrite the current proposition and change test coverage.

Should this be configurable?

Good question, in the #460 task there was only info that new API should be used when available and fall back to configmaps if not, so I didn't customize that

What kind of customization level you were thinking about?

User should be able to provide own NewResourceLock func? Like in this PR ✨ expose NewResourceLock in manager.Options #535?
Add field LeaderElectionLockType which would be an enum:
- EndpointsResourceLock
- ConfigMapsResourceLock
- LeasesResourceLock
  If the field will be not set by a user, then we set preferred one, for example, https://github.com/mszostok/controller-runtime/pull/1/files
both 1&2

one thing that comes to my mind which I want to clarify. This impl will be a little "breaking change", right?

Scenario 1:
User has a controller with old impl version, then updates the controller-runtime library. In the previous version the CM was used, in the new one, the Lease will be used. Problem is that the old ConfigMap will not be garbage collected, so we will have orphaned object

Scenario 2:
User updates the controller-runtime library and performs the rolling update of controller Deployment. Problem:

The previous version has an old strategy (ConfigMap) - still deployed
The new one has a new strategy (Lease) - newly deployed

Both will work at the same time (reconcile process). Because the old one will check the ConfigMap and the new one will check the Lease. So user needs to know controller Deployment should have the Recreate strategy to prevent such a situation.

So what about the migration path? Should it be automated? Documented?

fejta-bot · 2019-12-20T19:51:26Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

k8s-ci-robot · 2019-12-20T19:51:33Z

@mszostok: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2020-01-19T20:37:28Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

k8s-ci-robot · 2020-01-22T20:33:49Z

@mszostok: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-controller-runtime-test-master	`179f36f`	link	`/test pull-controller-runtime-test-master`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Fix some phrases in what_is_a_controller.md

DirectXMan12 · 2020-02-05T22:17:55Z

cc @JoelSpeed (not sure how active you want to be anymore -- lmk if you don't want to be tagged), @GrigoriyMikhalkin, @alvaroaleman (all of whom have had thoughts on LE in the past)

JoelSpeed · 2020-02-06T14:36:10Z

This seems sensible to me, glad someone's gotten around to doing this!

What happens during rolling updates when changing from the ConfigMap to ResourceLock implementations? I guess it's pretty unavoidable that the leader election will fail temporarily if two different versions of the controller are running?

fejta-bot · 2020-03-22T18:10:05Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-03-22T18:10:12Z

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

leongor · 2020-05-14T11:41:33Z

Is this PR still relevant? Can it be merged?

alvaroaleman · 2020-05-14T12:37:49Z

As Joel mentioned, we need to think about how we migrate existing controllers that currently use a configmap

leongor · 2020-05-14T15:31:37Z

I understand, but meantime this option can be used by those who do not need to migrate.
In our environment, it's needed desperately as we cannot use configmap.

alvaroaleman · 2020-05-14T15:45:39Z

I understand, but meantime this option can be used by those who do not need to migrate.
In our environment, it's needed desperately as we cannot use configmap.

Unfortunately not, because with the changes in this PR controllers will automatically use the lease lock if its available. This means that if a controller gets updated to this version of c-r and then deployed, there are two replicas that use a different mechanism for leader election which means they can hold a leader lock simultaneously. This means we very silently break valid expectations ("If I enable leader election, there will always at most be one active replica") and that is not an option.

Someone has to come up with a concept for what we can do to prevent that. Possible approaches include:

Default to configmap and only use leader lock if explicitly requested
Check if there is an existing configmap leader lock in the cluster and if yes, default to using configmap

leongor · 2020-05-14T15:54:09Z

I think it's a reasonable approach to stick with configmap as default and provide an option to change to lease.
The only thing that prevent to use lease today is that newResourceLock field in manager options is private and cannot be overridden.

alvaroaleman · 2020-05-14T16:03:33Z

Yeah, generally that sounds reasonable. It is subpar insofar that we basically say the recommended leader lock is not the default one due to these compatibility concerns but I guess its the best we can do and we can add a TODO to the manager to change the default when we break its interface.

leongor · 2020-05-14T18:37:12Z

Sounds good for me.

leongor · 2020-05-17T10:26:09Z

So, can we reopen this PR or do we need a new one?

alvaroaleman · 2020-05-17T14:50:54Z

Well, the PR got closed due to staleness which seems to indicate that the original author doesn't have the cycles to finish this up so opening a new PR may make sense.

…

On Sun, May 17, 2020 at 6:26 AM Leonid Gorelik ***@***.***> wrote: So, can we reopen this PR or do we need a new one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#600 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRR6ZBWUKBRZBDWDT4FTRDRR63U3ANCNFSM4IVXBN4Q> .

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 11, 2019

k8s-ci-robot requested review from droot and pwittrock September 11, 2019 17:56

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 11, 2019

mszostok force-pushed the myk-leader-election branch 2 times, most recently from 5e26c0e to fea2ea9 Compare September 11, 2019 18:44

k8s-ci-robot requested a review from DirectXMan12 September 11, 2019 19:05

klaudiagrz reviewed Sep 12, 2019

View reviewed changes

mszostok force-pushed the myk-leader-election branch 2 times, most recently from 8efda4e to 13fe693 Compare September 12, 2019 12:55

Add option to choose Lease lock if coordination group is available

179f36f

- Choose the Lease lock if lease.coordination.k8s.io is available otherwise, use ConfigMaps. - Add tests coverage for implemented logic using ginko & gomega - Remove outdated information about cert generation

mszostok force-pushed the myk-leader-election branch from 13fe693 to 179f36f Compare September 12, 2019 13:04

DirectXMan12 suggested changes Sep 17, 2019

View reviewed changes

DirectXMan12 mentioned this pull request Sep 17, 2019

✨ expose NewResourceLock in manager.Options #535

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 20, 2019

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 20, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 19, 2020

DirectXMan12 pushed a commit that referenced this pull request Jan 31, 2020

Merge pull request #600 from montaro/patch-1

d32b374

Fix some phrases in what_is_a_controller.md

vincepri added this to the Next milestone Feb 21, 2020

k8s-ci-robot closed this Mar 22, 2020

timebertt mentioned this pull request Aug 4, 2020

use leader election API if available #460

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨Add option to use Lease lock if coordination group is available #600

✨Add option to use Lease lock if coordination group is available #600

mszostok commented Sep 11, 2019 •

edited

k8s-ci-robot commented Sep 11, 2019

mszostok commented Sep 11, 2019

mszostok commented Sep 12, 2019 •

edited

DirectXMan12 left a comment

mszostok commented Sep 21, 2019 •

edited

fejta-bot commented Dec 20, 2019

k8s-ci-robot commented Dec 20, 2019

fejta-bot commented Jan 19, 2020

k8s-ci-robot commented Jan 22, 2020

DirectXMan12 commented Feb 5, 2020

JoelSpeed commented Feb 6, 2020

fejta-bot commented Mar 22, 2020

k8s-ci-robot commented Mar 22, 2020

leongor commented May 14, 2020

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020 •

edited

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020

leongor commented May 17, 2020

alvaroaleman commented May 17, 2020 via email

✨Add option to use Lease lock if coordination group is available #600

✨Add option to use Lease lock if coordination group is available #600

Conversation

mszostok commented Sep 11, 2019 • edited

k8s-ci-robot commented Sep 11, 2019

mszostok commented Sep 11, 2019

mszostok commented Sep 12, 2019 • edited

DirectXMan12 left a comment

Choose a reason for hiding this comment

mszostok commented Sep 21, 2019 • edited

fejta-bot commented Dec 20, 2019

k8s-ci-robot commented Dec 20, 2019

fejta-bot commented Jan 19, 2020

k8s-ci-robot commented Jan 22, 2020

DirectXMan12 commented Feb 5, 2020

JoelSpeed commented Feb 6, 2020

fejta-bot commented Mar 22, 2020

k8s-ci-robot commented Mar 22, 2020

leongor commented May 14, 2020

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020 • edited

alvaroaleman commented May 14, 2020

leongor commented May 14, 2020

leongor commented May 17, 2020

alvaroaleman commented May 17, 2020 via email

mszostok commented Sep 11, 2019 •

edited

mszostok commented Sep 12, 2019 •

edited

mszostok commented Sep 21, 2019 •

edited

leongor commented May 14, 2020 •

edited