-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
KEP: Finalizer Protection for Service LoadBalancers
- Loading branch information
Showing
1 changed file
with
156 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| --- | ||
| title: Finalizer Protection for Service LoadBalancers | ||
| authors: | ||
| - "@MrHohn" | ||
| owning-sig: sig-network | ||
| participating-sigs: | ||
| - sig-cloud-provider | ||
| reviewers: | ||
| - "@andrewsykim" | ||
| - "@bowei" | ||
| - "@jhorwit2" | ||
| - "@jiatongw" | ||
| approvers: | ||
| - "@andrewsykim" | ||
| - "@bowei" | ||
| - "@thockin" | ||
| editor: TBD | ||
| creation-date: 2019-04-23 | ||
| last-updated: 2019-04-23 | ||
| status: provisional | ||
| see-also: | ||
| replaces: | ||
| superseded-by: | ||
| --- | ||
|
|
||
| # Finalizer Protection for Service LoadBalancers | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Title](#title) | ||
| - [Table of Contents](#table-of-contents) | ||
| - [Release Signoff Checklist](#release-signoff-checklist) | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Goals](#goals) | ||
| - [Proposal](#proposal) | ||
| - [Test Plan](#test-plan) | ||
| - [Graduation Criteria](#graduation-criteria) | ||
| - [Implementation History](#implementation-history) | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR) | ||
| - [ ] KEP approvers have set the KEP status to `implementable` | ||
| - [ ] Design details are appropriately documented | ||
| - [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input | ||
| - [ ] Graduation criteria is in place | ||
| - [ ] "Implementation History" section is up-to-date for milestone | ||
| - [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] | ||
| - [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes | ||
|
|
||
| ## Summary | ||
|
|
||
| We will be adding finalizer protection to ensure the Service resource is not | ||
| fully deleted until the correlating load balancing resources are also deleted. | ||
| Any service that has `type=LoadBalancer` (both existing and newly created ones) | ||
| will be attached a service LoadBalancer finalizer, which should be removed by | ||
| service controller upon the cleanup of related load balancing resources. Such | ||
| finalizer protection mechanism will be released with phases to ensure downgrades | ||
| can happen safely. | ||
|
|
||
| ## Motivation | ||
|
|
||
| There are various cases where service controller can leave orphaned load | ||
| balancing resources after services are deleted (ref discussion on | ||
| https://github.com/kubernetes/kubernetes/issues/32157, | ||
| https://github.com/kubernetes/kubernetes/issues/53451). We are periodically | ||
| getting bug reports and customer issues that replicated such problem, which | ||
| seems to be common enough and is worth to have a better mechanism for ensuring | ||
| the cleanup of load balancing resources. | ||
|
|
||
| ### Goals | ||
|
|
||
| Ensure the Service resource is not fully deleted until the correlating load | ||
| balancing resources are also deleted. | ||
|
|
||
| ## Proposal | ||
|
|
||
| We are going to define a finalizer for service LoadBalancers with name | ||
| `kubernetes.io/service-load-balancer`. This finalizer will be attached to any | ||
| service that has `type=LoadBalancer`. Upon the deletion of such servicce, the | ||
| actual deletion of the resource will be blocked until this finalizer is removed. | ||
| This finalizer will not be removed until cleanup of the correlating load | ||
| balancing resources are considered finished by service controller. | ||
|
|
||
| Note that the removal of this finalizer might also happen when service type | ||
| changes from `LoadBalancer` to another. This however doesn't change the | ||
| implication that the resources cleanup must be fulfilled before fully deleting | ||
| the service. | ||
|
|
||
| The lifecyle of a `LoadBalancer` type serivce with finalizer would look like: | ||
| - Creation | ||
| 1. User creates a service. | ||
| 2. Service controller observes the creation and attaches finalizer to the service. | ||
| 3. Provision of load balancing resources. | ||
| - Deletion | ||
| 1. User issues a deletion for the service. | ||
| 2. Service resoure deletion is blocked due to the finalizer. | ||
| 3. Service controller observed the deletion timestamp is added. | ||
| 4. Cleanup of load balancing resources. | ||
| 5. Service controller removes finalizer from the service. | ||
| 6. Service resource deleted. | ||
| - Update to another type | ||
| 1. User update service from `type=LoadBalancer` to another. | ||
| 2. Service controller observed the update. | ||
| 3. Cleanup of load balancing resources. | ||
| 4. Service controller removes finalizer from the service. | ||
|
|
||
| The expected cluster upgrade/downgrade path for service with finalizer would be: | ||
| - Upgrade from pre-finalizer version | ||
| - All existing `LoadBalancer` services will be attached a finalzer upon startup | ||
| of the new version of service controller. | ||
| - The newly created `LoadBalancer` services will have finalizer attached upon | ||
| creation. | ||
| - Downgrade from with-finailzer version | ||
| - All existing `LoadBalancer` service will have the attached finalizer removed | ||
| upon the cleanup of load balancing resources. | ||
| - The newly created `LoadBalancer` services will not have finailzer attached. | ||
|
|
||
| To ensures that downgrades can happen safely, the first release will include the | ||
| "remove finalizer" logic with the "add finalizer" logic behind a gate. Then in a | ||
| later release we will remove the feature gate and enable both the "remove" and | ||
| "add" logic by default. | ||
|
|
||
| As such, we are proposing Alpha/Beta/GA phase for this enhancement as below: | ||
| - Alpha: The finailizer cleanup logic would be on by default while the finalizer | ||
| addition logic will be put behind a feature gate that is off by default. | ||
| - Beta: Both the cleanup and addition logic will be default on, but allowing | ||
| disabling via feature gate. | ||
| - GA: Service LoadBalancers Finalizer Protection will always be on. | ||
|
|
||
| ### Test Plan | ||
|
|
||
| We will implement e2e test cases to ensure: | ||
| - Service finalizer protection works with various service lifecycles on a cluster | ||
| that enables it. | ||
|
|
||
| In addition to above, we should have upgrade/downgrade tests that: | ||
| - Verify the downgrade path and ensure service finalizer removal works. | ||
| - Verify the upgrade path and ensure finalizer protection works with existing LB | ||
| services. | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| Beta: Allow Alpha ("remove finalzier") to soak for at least one release, then | ||
| switch the "add finalizer" logic to be on by default. | ||
|
|
||
| GA: TBD | ||
|
|
||
| ## Implementation History | ||
|
|
||
| - 2017-10-25 - First attempt of adding finalizer to service | ||
| (https://github.com/kubernetes/kubernetes/pull/54569) | ||
| - 2018-07-06 - Split finalizer cleanup logic to a separate PR | ||
| (https://github.com/kubernetes/kubernetes/pull/65912) | ||
| - 2019-04-23 - Creation of the KEP |