Fix kubelet deadlock

[bprashanth]

1. User deletes all multi container pods on a node at once, after the containers have started
2. All pods are synced for each delete notification
3. Each pod sends a status update because of the bug in my first commit
4. Status manager's podStatusChannel fills up
5. An update fails becuase a pod in the channel has already been deleted from the apiserver
6. Status manager deadlock
7. SyncPods enters the deadlock next time it tries to talk to the status manager

@dchen1107

[k8s-bot]

GCE e2e build/test failed for commit 9e1ab43c2fb279a7f9cc28885d69ed03edbdf611.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[dchen1107]

Ahh, this is the deadlock issue I suspected and planed to look into, but distracted by all those p0 issues. Thanks, but I don't have time to review it today. Assigned it to @yujuhong. YuJu, here is a little bit background:
we found sometimes when currently deleting multiple pods, and each pods with multiple containers, kubelet might run into a deadlock. The issue was originally reported by #9833. Currently we have a proposed workaround #10004 to detect such deadlock, and restart kubelet. But that workaround should be the last resort. Thanks!

cc/ @saad-ali and @ArtfulCoder too.

[k8s-bot]

GCE e2e build/test passed for commit 5f61392.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[yujuhong]

LGTM overall with a nit/question.

[bprashanth]

Added equality check in status manager and a unittests. PTAL.

[k8s-bot]

GCE e2e build/test failed for commit 9680a1293207b4f42c1e2429473ddf00c85953f3.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[yujuhong]

LGTM overall. e2e failed though.

Also, @dchen1107 for the second LGTM.

[bprashanth]

Rerunning failed test on local cluster, pushed anew to get a normal run as well since it passed the previous time so I'm suspecting it's unrelated

[yujuhong]

It's probably not related, as I just saw a similar failure on another PR.

[k8s-bot]

GCE e2e build/test passed for commit af175cc.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[bprashanth]

Risk assessment: medium-low. Code sorts a list of containers and shoves a Delete into a goroutine.
Value: the kubelet doesn't deadlock or flood the apiserver with updates when it has multi-container pods.

@dchen1107 PTAL, I ran the previous run's failing container probe e2e locally and got 4/4 pass

[dchen1107]

LGTM. @davidopp Can we have your LGTM on this one? Thanks!

[davidopp]

LGTM