[kube-up][scale-out] coredns pod may sometimes be stuck in "ContainerCreating" for newly created tenant

[h-w-chen]

What happened:
coredns pod of new tenant's default network(vpc) may be stuck in ContainerCreating status:

$ kubectl get pod -AT -owide
...
x26      kube-system   coredns-default-hwc-kubeup-test-tp-1-7456994c79-z58bx   8476672419923472858   0/1     ContainerCreating   0          11m   <none>         hwc-kubeup-test-rp-1-minion-group-z9z4   <none>           <none>

What you expected to happen:
coredns pod in Running state

How to reproduce it (as minimally and precisely as possible):
at a kube-up scale-out 1tp 1rp 1 worker cluster, login to tp master, create multiple tenants, then look at the pods

kubectl create tenant x1
kubectl create tenant x2
...
kubectl get pod -AT -owide

Anything else we need to know?:
likely we have to create 30-40 tenants to catch a few such issues.

Environment:

• Arktos version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools: kube-up 1tp 1rp 1worker scale-out
• Network plugin and version (if this is a network-related bug): mizar
• Others:

[h-w-chen]

it seems deleting the pod and the auto-provisioned one still be stuck in ContainerCreating

[Sindica]

Sonya saw one occurence for kube-up scale up on 3/10 in master. This should not be a launch blocker as we are targeting kube-up scale out only.