Add Service Load Balancer finalizer support

[MrHohn]

What type of PR is this?
/kind feature

What this PR does / why we need it:
KEP link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20190423-service-lb-finalizer.md

This PR adds finalizer protection for service LoadBalancers. It defines an alpha feature gate for the finalizer addition part while the removal part is always on.

Note that is is largely based off of #54569 and #65912. While one significant difference is that I haven't removed the cached service layer given the complexity and risks of hosting both pre-finalizer and with-finalizer control flow logic. Keeping the cache layer makes this easier.

End-to-end tests is being added in #78410.

Which issue(s) this PR fixes:

Fixes #53451. Also ref kubernetes/enhancements#980 and kubernetes/cloud-provider#16.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Finalizer Protection for Service LoadBalancers is now added as Alpha (disabled by default). This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted.

[MrHohn]

/test all

[MrHohn]

So I'm having trouble with unit testing:

--- FAIL: TestSyncLoadBalancerIfNeeded (0.06s)
    --- FAIL: TestSyncLoadBalancerIfNeeded/service_with_finalizer_that_no_longer_wants_LB (0.00s)
        service_controller_test.go:319: Service hasFinalizer=true, want false
    --- FAIL: TestSyncLoadBalancerIfNeeded/service_that_needs_cleanup (0.00s)
        service_controller_test.go:319: Service hasFinalizer=true, want false
--- FAIL: TestPatchFinalizer (0.00s)
    --- FAIL: TestPatchFinalizer/remove_finalizer (0.00s)
        service_controller_test.go:1136: Service hasFinalizer = true, want false
--- FAIL: TestPatchStatus (0.00s)
    --- FAIL: TestPatchStatus/clear_status (0.00s)
        service_controller_test.go:1214: Got status {[{8.8.8.8 }]}, want &LoadBalancerStatus{Ingress:[],}
FAIL

It seems like the fake kube client doesn't properly support directive delete for patch operation, hence all the test cases that require removing fields failed. As I can confirm the generated patch bytes indeed contain directive:

{"metadata":{"$deleteFromPrimitiveList/finalizers":["service.kubernetes.io/load-balancer-cleanup"],"$setElementOrder/finalizers":["bar"],"finalizers":["bar"]}}

My current plan is fixing that fake kube client.

[MrHohn]

@andrewsykim Sorry, I rebased to fix the merge conflict in kube_features.go. Would be great to have a re-lgtm, thanks!

[andrewsykim]

/lgtm

[MrHohn]

/retest

[timoreimann]

@MrHohn apologies for commenting on this merged PR with a late thought:

Is there a need to consider the edge case where the type is changed from LoadBalancer, the LB resource is about to be cleaned up, and while that process is still ongoing (possibly delayed/retried due to API connectivity issues or other events), the type is being switched back to LoadBalancer again? Would we then cancel the deletion efforts (or delete and recreate in sequence)?

It didn't become immediately clear to me by looking at the change, but my familiarity with the code is limited.

[bowei]

While the object is in a finalization state, I would imagine changing the fields in the object should not have any effect. In other words, the user would have to wait for the finalizers to be done processing and the object has been fully removed before recreating the object.

[MrHohn]

@timoreimann No worries at all and thanks for giving thoughts on this.

For the case you mentioned (type changed from LoadBalancer then back), there will be two outcomes depends on the timeline.

The first outcome is "Delete and Recreate". It happens if service controller starts processing the type:LoadBalancer->other update before it received the type:other->LoadBalancer update.

The second outcome is "No-op or Reconciling". It happens if service controller hasn't got to process the type:LoadBalancer->other update before the type:other->LoadBalancer update comes. So service controller processes both two updates together. In this case service controller simply calls EnsureLoadBalancer() again. It can either be a no-op if nothing changed, or a reconciliation if there is other valid update.

[MrHohn]

While the object is in a finalization state, I would imagine changing the fields in the object should not have any effect.

Note that in this case, the behavior in fact stays the same as before this finalizer support. As the LB deletion action is triggered by a type change update, instead of an object deletion with finalizer.

But yes, for the cases where the object is in a finalization state, update won't have any effect.

[timoreimann]

@MrHohn @bowei thanks for explaining -- happy to hear we're able to cope with the described scenario just fine. 👍

[sylr]

If proven effective could this be backported til 1.13 ?

[MrHohn]

@sylr Unfortunately it seems less likely we will backport a feature to 1.13.

[andrewsykim]

Backporting to v1.13 will also break backwards compatibility for the finalizer in v1.12.