[Federation] Daemonset with finalizer not always getting deleted

[csbell]

From CI failure here: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2458/build-log.txt

Upon deleting with orphan=true, the finalizer Update() fails and the next time around the federated object is never deleted. This is flaky.

Relevant bits:

I0329 11:50:35.083] I0329 18:49:17.185376       1 daemonset_controller.go:273] Trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.185529       1 daemonset_controller.go:338] Reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.185667       1 daemonset_controller.go:462] Handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 149 2 2017-03-29 18:49:11 +0000 UTC 2017-03-29 18:49:16 +0000 UTC 0xc4229b4398 map[aaa:bbb] map[] [] [federation.kubernetes.io/delete-from-underlying-clusters orphan] } {&LabelSelector{MatchLabels:map[string]string{aaa: bbb,},MatchExpressions:[],} {{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[aaa:bbb] map[] [] [] } {[] [] [{container1 gcr.io/google_containers/serve_hostname:v1.4 [] []  [{ 0 9376 TCP }] [] [] {map[] map[]} [] nil nil nil /dev/termination-log File IfNotPresent nil false false false}] Always 0xc4229b43b0 <nil> ClusterFirst map[]   <nil>  false false false &PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,} []   nil default-scheduler []}} {OnDelete nil} 0 1} {0 0 0 0 0 0 0 0}}
I0329 11:50:35.084] I0329 18:49:17.185966       1 deletion_helper.go:115] Handling deletion of federated dependents for object: federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.186060       1 deletion_helper.go:122] Found finalizer orphan. Nothing to do, just remove the finalizer
I0329 11:50:35.085] E0329 18:49:17.215415       1 daemonset_controller.go:368] Failed to delete federated-daemonset: failed to remove finalizer orphan from daemonset : Operation cannot be fulfilled on daemonsets.extensions "federated-daemonset": the object has been modified; please apply your changes to the latest version and try again
I0329 11:50:35.086] I0329 18:49:17.219857       1 daemonset_controller.go:273] Trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.219936       1 daemonset_controller.go:338] Reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.220059       1 daemonset_controller.go:462] Handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 154 2 2017-03-29 18:49:11 +0000 UTC 2017-03-29 18:49:16 +0000 UTC 0xc422990c58 map[aaa:bbb] map[] [] [orphan] } {&LabelSelector{MatchLabels:map[string]string{aaa: bbb,},MatchExpressions:[],} {{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[aaa:bbb] map[] [] [] } {[] [] [{container1 gcr.io/google_containers/serve_hostname:v1.4 [] []  [{ 0 9376 TCP }] [] [] {map[] map[]} [] nil nil nil /dev/termination-log File IfNotPresent nil false false false}] Always 0xc422990c80 <nil> ClusterFirst map[]   <nil>  false false false &PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,} []   nil default-scheduler []}} {OnDelete nil} 0 1} {0 0 0 0 0 0 0 0}}
I0329 11:50:35.086] I0329 18:49:17.220332       1 deletion_helper.go:115] Handling deletion of federated dependents for object: federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.220420       1 deletion_helper.go:117] obj does not have federation.kubernetes.io/delete-from-underlying-clusters finalizer. Nothing to do

[csbell]

Discussed this with Nikhil earlier today. The buggy behavior is a result of handling removal of 2 or more annotations. In this case, the daemonset and namespace controllers both try to remove orphan and delete-from finalizers but the code isn't written to expect separate updates.

Seen again here today: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2570

Marking P0. With the fix in, we should be above 90% in no-flakes.

[nikhiljindal]

Sent #44084 to fix this