fix registry addon on Docker Driver with Containerd Runtime

[medyagh]

a follow up to this PR
#10782
and this issue #10778

the PR #10782 fixed VM by enabling portmap the Bridge.
but KIC still has problem because sometimes Kindnet does not get picked up (do to the naming)

VM:

$ ls /etc/cni/net.d/
1-k8s.conf  87-podman-bridge.conflist

Kic:

docker@minikube:~$ ls /etc/cni/net.d/
1-k8s.conf  100-crio-bridge.conf  200-loopback.conf  87-podman-bridge.conflist

The error can be seen in gopogh report
https://storage.googleapis.com/minikube-builds/logs/10782/71145bc/Docker_Linux_containerd.html#fail_TestAddons%2fparallel%2fRegistry

important slack chat with @afbjorklund

https://kubernetes.slack.com/archives/C1F5CT6Q1/p1615410452004400

[afbjorklund]

I don't get bridge for KIC when running with CNI, it tries to use kindnet...
But kubernetes gets started before, so there is a chicken-and-egg issue.

minikube start --driver docker --container-runtime containerd

📦 Preparing Kubernetes v1.20.2 on containerd 1.4.3 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...

At boot:

docker@minikube:~$ ls /etc/cni/net.d/
100-crio-bridge.conf  200-loopback.conf  87-podman-bridge.conflist

After a while:

docker@minikube:~$ ls /etc/cni/net.d/
10-kindnet.conflist   200-loopback.conf
100-crio-bridge.conf  87-podman-bridge.conflist

Even after we remove the unused cri-o CNI, we still have podman CNI
(which isn't used by Kubernetes at all, since it is only used by Podman!)

So it gets the same problem as #10384

Currently we don't install /etc/cni/net.d/10-containerd-net.conflist
https://github.com/containerd/containerd/blob/master/script/setup/install-cni

---

For cri-o we did a hack for this, to restart the CRI after it had started up.
Another hack is to remove the CNI configuration while booting kube-proxy.

The best solution would be if Kubernetes had a flag to choose CNI name...

Instead of the current stupid rule:

If there are multiple CNI configuration files in the directory, the kubelet
uses the configuration file that comes first by name in lexicographic order.

https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni

So I guess we have to clear out the common system /etc/cni/net.d directory,
wait for Kubernetes + CNI to be be booted, and then restore the old directory again.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten