CNS hands out duplicate IP's to CNI in race condition

[matmerr]

What happened:

goldpinger-deploy-8f8746dbb-qq6h5   1/1     Running   2          5d      10.241.0.22    aks-nodepool1-25205300-vmss000001   <none>           <none>
goldpinger-deploy-8f8746dbb-wv564   1/1     Running   2          5d      10.241.0.22    aks-nodepool1-25205300-vmss000001   <none>           <none>

There are two pods running at the same time with duplicate IP's.

What you expected to happen:

How to reproduce it:

When CNI calls CNS to free an IP, CNI only presents the name of the pod and pod namespace to CNS to free the IP.

Race condition can happen in this scenario:

1. Containerd restart.
2. Tries to remove all previous containers but there's lockcontention so it takes 10's of seconds
3. Tries to add all new containers
4. One of the adds gets the lock before one of the deletes cause there is no fair queuing.
5. We delete a valid running container

Orchestrator and Version (e.g. Kubernetes, Docker):

Operating System (Linux/Windows):

Kernel (e.g. uanme -a for Linux or $(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion for Windows):

Anything else we need to know?:
[Miscellaneous information that will assist in solving the issue.]

[rbtr]

closed in #875