Unable to start minikube with none driver with Cilium on Ubuntu 21.10

[matihost]

Pointing here, but maybe it is something related to Cilium..

After upgrading to Ubuntu 21.10 (from 21.04)

sudo -E minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --kubernetes-version=stable --extra-config=apiserver.enable-admission-plugins=NamespaceExists --extra-config=kubelet.resolv-conf=/tmp/resolv.conf --extra-config=kubelet.cgroup-driver=systemd --addons=pod-security-policy --network-plugin=cni

helm install cilium cilium/cilium --namespace kube-system --set operator.replicas=1

(where /tmp/resolv.conf contains nameserver 8.8.8.8 only)

After cilium start, the coredns pod is unable to reach forwarding dns:

 [ERROR] plugin/errors: 2 3613863210877316009.6342293437640210488. HINFO: read udp 172.26.0.153:56484->8.8.8.8:53: i/o timeout 

and indeed no pod from managed by Cilium is unable to reach anything. Pod cilium and any hostNetwork pod is fine.

Routing table and iptables seems fine.
Custom pod cidr --set ipam.operator.clusterPoolIPv4PodCIDR="172.26.0.0/16" or use some previous version dint hepl --version 1.10.1

Also under minikube containerd (--container-runtime=containerd --network-plugin=cilium added to minikube cmd) - is not working, but there cilium is failing and ability to connect to k8s api server is lost.

Running minikube w/ none /w docker but w/o --network-plugin=cni works fine.

Attached minikube logs for minikube_with_docker_with_cni_cilium.logs.txt
and
minikube_with_containerd_with_cilium.logs.txt

[spowelljr]

Hi @matihost, thanks for reporting your issue with minikube!

Could you please run minikube logs --file=logs.txt and upload the logs.txt file to this issue so we can use the logs to debug, thanks!

[matihost]

Hi, updated original entry with logs.

[matihost]

Per Cilium k8s 1.21.x is only supported

The default docker driver works:

minikube start --network-plugin=cni --kubernetes-version=1.21.6
helm install cilium cilium/cilium --namespace kube-system --set operator.replicas=1

so it seems it is related to none minikube driver and upgraded ubuntu

[sharifelgamal]

it's worth mentioning that we also have a more user-friendly --cni flag, that allows you to just pass in --cni=cilium or a path to a more specific yaml file you want to apply. not sure how helpful that is in this particular case, but still useful information nonetheless.

this strikes me as an issue with the combo of ubuntu 21.10 and cilium, but it's hard to be sure. i'll keep this issue open in case anyone else has better insight.

[matihost]

with -cni=cilium same effect:

sudo -E minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --kubernetes-version=stable --extra-config=apiserver.enable-admission-plugins=NamespaceExists --extra-config=kubelet.resolv-conf=/tmp/resolv.conf --extra-config=kubelet.cgroup-driver=systemd --addons=pod-security-policy --cni=cilium

core-dns errors:

│ [ERROR] plugin/errors: 2 6799450272274576085.9022900702037546578. HINFO: read udp 10.244.0.165:57498->8.8.8.8:53: i/o timeout                                                                            │
│ [ERROR] plugin/errors: 2 6799450272274576085.9022900702037546578. HINFO: read udp 10.244.0.165:47002->8.8.8.8:53: i/o timeout   

[kubealex]

same here and also on non-minikube setup on 1.23 and 1.24. Ubuntu 21.10.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale