Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
kubeadm 1.6.0 (only 1.6.0!!) is broken due to unconfigured CNI making kubelet NotReady #43815
Initial report in kubernetes/kubeadm#212.
I suspect that this was introduced in #43474.
What is going on (all on single master):
In the conditions list for the node:
Previous behavior was for the kubelet to join the cluster even with unconfigured CNI. The user will then typically run a DaemonSet with host networking to bootstrap CNI on all nodes. The fact that the node never joins means that, fundamentally, DaemonSets cannot be used to bootstrap CNI.
Edit by @mikedanese: please test patched debian amd64 kubeadm #43815 (comment) with fix
It looks like DaemonSets will still get scheduled even if the node is not ready. This is really, in this case,
The current plan that we are going to test out is to have
@kensimon is testing this out.
I have kubeadm getting past the node's
I'm exploring some other options like not marking the node as notReady, but it's not clear that's what we want to do.
@overip you need to edit /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS
and then restart kubelet after that kubeadm init should work.
All correct, and while we're at it
If you see this:
you have to edit your /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add the flag --cgroup-driver="systemd"
and do as above
I also got the same issue,but I just copy the cni config on the master node to the corresponding location of the worker node,then it became OK.
systemctl status kubelet.service -l
● kubelet.service - kubelet: The Kubernetes Node Agent
Jun 06 10:59:46 contiv1.com kubelet: W0606 10:59:46.215827 4414 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
The status of all nodes:
Being new to setting up Kubernetes I get superconfused. I tried following https://medium.com/@SystemMining/setup-kubenetes-cluster-on-ubuntu-16-04-with-kubeadm-336f4061d929 that use weave-kube for network but I'm also stuck with the same issue. Any way to solve this?
Why is this still an issue? Ubuntu 16.04/CentOS 7.3 with latest updates using the official k8s repos with 1.6.4 and following the steps outlined here: https://kubernetes.io/docs/setup/independent/install-kubeadm/
@drajen No, this affected only v1.6.0. It's expected that kubelet doesn't find a network since you haven't installed any. For example, just run
to install Weave Net and those problems will go away. You can choose to install Flannel, Calico, Canal or whatever CNI network if you'd like
@drajen I think @luxas' point is that this is the wrong place to be asking about setup.
At least for CentOS/RHEL, make sure you update /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add the flag --cgroup-driver="systemd"
If you reinstall again on the same machine, this is a full proper reset:
I hit this issue, and absolutely nothing I read above worked. So I tried again with a much more controlled setup, switching from Ubuntu to the latest CoreOS, going to an earlier version of k8s to start with, and in general being very anal about every last thing installed into each VM. I am NOT using kubeadm, but instead a combination of vagrant and ansible.
(why? because I had no idea what was going on in kubeadm and figured this way at least I'd have control and be able to bypass any overzealous preflight checks, not to mention feeling like I had more automation control in general, and also not having to worry about the warning to do-not-apply-alpha-software-in-production.)
When I tried this setup with an older (1.4.3) edition of k8s, this approach was golden. I then tried upgrading to 1.71. Once again, I am STILL hitting this same issue in spite of using no kubeadm at all.
I have confirmed that I am running calico in each of my nodes, including the Master and the three potential Workers. ALL of my nodes are reporting as NotReady, so I'm not really sure how I could apply weave (or anything else) to get it running.
This whole thing just seems chicken/egg ... can't allocate a pod because networking is failing, but need networking running to create a network at /etc/cni/net.d in order to be able to allocate pods. And again, all this was working a few hours ago with k8s 1.4.3. I'm very frustrated!
I'd appreciate any insights anybody could give.
On master: journalctl -r -u kubelet gives me
Jul 24 00:48:16 rogue-kube-master-01 kubelet-wrapper: E0724 00:48:16.592274 7647 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is no
docker ps | grep calico
(truncated for readability)
There is no /etc/cni/net.d
From my kubectl box:
kubectl apply -f https://git.io/weave-kube-1.6
DID NOT fix anything and in fact only seems to create more problems.
bill@rogue:~/vagrant_rogue/rogue-cluster$ kubectl apply -f https://git.io/weave-kube-1.6
bill@rogue:~/vagrant_rogue/rogue-cluster$ kubectl get pods --namespace=kube-system
Deeper investigation of the weave errors indicate that they either (a) cannot connect to the apiserver, or else (b) in the case of the one marked "Running" it's complaining that it cannot connect to itself.
@luxas Respectfully, I have to question whether this is a new issue. Since I am getting the exact same result setting up k8s without kubeadm as everyone else is getting with kubeadm, this seems to eliminate kubeadm as the source of the problem. Perhaps this issue ought to be renamed accordingly?
@billmilligan respectfully, since the issue is about kubeadm, and your able to reproduce it without kubeadm, then it is the wrong place to file it? I think this thread solved the kubeadm issue. This is a new issue. I think it will get more attention as a new issue. As people on this thread think its already solved and are ignoring it.
I use kubeadm and was effected by this issue, and it has been solved since 1.6.1. I've deployed losts of k8s's since, so I really do think you have a separate issue.
The error message
is NOT necessarily bad.
That error message tells you that you have to plugin in a third party CNI spec implementation provider.
What is CNI and how does integrate with Kubernetes?
CNI stands for Container Network Interface and defines a specification that kubelet uses for creating a network for the cluster. See this page for more information how Kubernetes uses the CNI spec to create a network for the cluster.
Kubernetes doesn't care how the network is created as long as it satisfies the CNI spec.
This means, that after
this is very much expected. If this wasn't the case, we would have favored a specific network provider.
So how do I "fix" this error?
The DaemonSet will copy out the CNI binaries needed to
After the CNI provider is installed, the kubelet will notice that "oh I have some information how to set up the network", and will use the 3rd-party configuration and binaries.
And when the network is set up by the 3rd-party provider (by kubelet invoking it), the Node will mark itself
How is this issue related to kubeadm?
Late in the v1.6 cycle, a PR was merged that changed the way kubelet reported the
kubeadm in v1.6.0 waited wrongly for the master node to be in the
THAT WAIT MISCONCEPTION ON THE KUBEADM SIDE IS WHAT THIS ISSUE IS ABOUT
However, we quickly fixed the regression in v1.6.1 and released it some days after v1.6.0.
Please read the retrospective for more information about this, and why v1.6.0 could be released with this flaw.
So, what do you do if you think you see this issue in kubeadm v1.6.1+?
Well, I really think you don't. This issue is about when
What you WILL see though is
Anyway, please open a new issue if you see something unexpected with kubeadm
Please do not comment more on this issue. Instead open a new one.
@billmilligan So you only have to
I'm pretty much summarizing what has been said above, but hopefully in a more clear and detailed way.
(Lastly, sorry for that much bold text, but I felt like it was needed to get people's attention.)