DaemonSet Controller doesn't delete orphaned pods

[krzysztof-jastrzebski]

What happened:
DaemonSet doesn't delete pods that:

• weren't scheduled yet,
• node was deleted so pod can't be scheduled.
  What you expected to happen:
  Pod should be deleted when node is removed.
  How to reproduce it (as minimally and precisely as possible):
  I encountered this problem when doing tests with cluster with 1000 nodes and 30000 pods running. During tests I was deleting and adding nodes using GCE API without draining them. I reproduced problem on much smaller cluster but it requires stopping K8s scheduler.

Scenario:

1. Create cluster with 2 nodes.
2. Wait till fluentd-gcp pods are running on nodes.
3. Stop K8s scheduler.
4. Delete fluentd-gcp pods.
5. Wait till daemonset controller creates pending pods.
6. Delete one node.
7. Start scheduler.
8. One of the fluentd-gcp pods will be pending and scheduler won't be able to schedule.

Anything else we need to know?:

Environment: GKE

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.2-gke.3", GitCommit:"3bdb4f77629a001276ae061e68aff4bac147fbc5", GitTreeState:"clean", BuildDate:"2018-11-13T22:54:51Z", GoVersion:"go1.10.4b4", Compiler:"gc", Platform:"linux/amd64"}

/kind bug

[krzysztof-jastrzebski]

/sig apps

[krzysztof-jastrzebski]

/assign janetkuo

[liggitt]

/assign @k82cn

[k82cn]

We have PodGC to clean up orphaned pods every 20s, is that enough?

[krzysztof-jastrzebski]

I'm removing pods that have only node affinity set and don't have pod.Spec.NodeName set. These pods were never touched by scheduler before node was removed and after node was removed can't be scheduled as node is doesn't exist anymore. Such pods are pending forever and PodGC can't remove them as pod.Spec.NodeName is empty.

[k82cn]

have only node affinity set and don't have pod.Spec.NodeName set

Yes, that's right :) let me take few time to review the PR, thanks.

[krzysztof-jastrzebski]

@k82cn Should I backport fix to 1.13 and 1.12?

[k82cn]

Should I backport fix to 1.13 and 1.12?

+1