Static pod terminating status can not be bring back

[xinxiaogang]

If the mirror pod of static pod be deleted via apiserver, the mirror pod status would stuck at Terminating. Even restarting kubelet does not work.

kubelet should sync the running static pod status to mirror pod.

[adohe-zz]

@kubernetes/sig-node

[Random-Liu]

@xinxiaogang Now kubelet ignores all modifications to the mirror pod as long as the annotation remains unchanged. There are more mirror pod related discussion in #16627, you can follow up if you are interested, :)

However, I think what you suggested is reasonable somehow.

@yujuhong except for the annotation change, if the Deletiontimestamp of the mirror pod is set, shouldn't we recreate it?

[xinxiaogang]

Thanks @Random-Liu . I have a PR for the fix and it works in our cluster now. #22626

Actually, all of our static pods (etcd, apiserver, etc) are running well, but the problem of Terminating stuck is, the endpoints would be picked out of my service and no traffic would be dispatched to these Terminating pods.

[Random-Liu]

@xinxiaogang Thank you very much for the fix. :) We'll review it.

[yujuhong]

@xinxiaogang, as a quick workaround, you can force delete the mirror pods (kubectl delete pod --grace-period=0 <pod_name>), and kubelet will recreate all of them.

@yujuhong except for the annotation change, if the Deletiontimestamp of the mirror pod is set, shouldn't we recreate it?

Yes, we should do it. Ideally we should resolve the mirror pod issue #16627 altogether, but this seems like a relatively isolated fix.

Thanks @Random-Liu . I have a PR for the fix and it works in our cluster now. #22626

Thanks for the PR!