New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network plugin is not ready: cni config uninitialized #48798

Closed
PLoic opened this Issue Jul 12, 2017 · 53 comments

Comments

Projects
None yet
@PLoic
Copy link

PLoic commented Jul 12, 2017

Hello, I want to do a fresh install of kubernetes via kubeadm, but when I start the install I'm stuck on

[apiclient] Created API client, waiting for the control plane to become ready

When I do a journalctl -xe I see :

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

And I don't why I get this error. I also tried to disable firewalld but no effect.

Environment:

  • Kubernetes version (use kubectl version): v1.7.0
  • Cloud provider or hardware configuration**:
  • OS (e.g. from /etc/os-release): CentOS 7
  • Kernel (e.g. uname -a): 3.10.0-514.26.2.el7.x86_64
  • Install tools: Kubeadm
  • Others:
    docker version : Docker version 17.06.0-ce, build 02c1d87
    My RPM version :

kubeadm-1.7.0, kubectl-1.7.0, kubelet-1.7.0, kubernetes-cni-0.5.1

Thanks for your help

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Jul 12, 2017

@PLoic There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

@PLoic

This comment has been minimized.

Copy link
Author

PLoic commented Jul 12, 2017

/area [kubeadm]

@dcbw

This comment has been minimized.

Copy link
Member

dcbw commented Jul 12, 2017

@PLoic you get this error because no CNI network has been defined in /etc/cni/net.d and you're apparently using the CNI network plugin. Something has to write a config file to that directory to tell the CNI driver how to configure networking. I'm not sure what/how kubeadm does that though, so I'll leave that to @jbeda or other kubeadm folks.

@xiangpengzhao

This comment has been minimized.

Copy link
Member

xiangpengzhao commented Jul 12, 2017

xref: #43567

@alioliver

This comment has been minimized.

Copy link

alioliver commented Jul 13, 2017

@dcbw Hi dcbw , Environment same to @PLoic , but I get this same error

@PLoic PLoic referenced this issue Jul 13, 2017

Closed

Kubeadm join fail #344

@PLoic

This comment has been minimized.

Copy link
Author

PLoic commented Jul 13, 2017

It's seem to working by removing the$KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

@tiendungitd

This comment has been minimized.

Copy link

tiendungitd commented Jul 13, 2017

removing $KUBELET_NETWORK_ARGS not work with me.

@alioliver

This comment has been minimized.

Copy link

alioliver commented Jul 13, 2017

@PLoic also not work with me

@bboreham

This comment has been minimized.

Copy link
Contributor

bboreham commented Jul 13, 2017

@PLoic at step 3 which pod network did you install? There are various choices, and troubleshooting after that depends on the specific case.

@dcbw

This comment has been minimized.

Copy link
Member

dcbw commented Jul 13, 2017

@PLoic also, kubelet logs would be great

@tiendungitd

This comment has been minimized.

Copy link

tiendungitd commented Jul 14, 2017

try to apply this plugin: kubectl apply --filename https://git.io/weave-kube-1.6
it works for me.

@ljb-2000

This comment has been minimized.

Copy link

ljb-2000 commented Jul 14, 2017

@PLoic @dcbw  I install flannel plugin of k8s(1.7) still get this same error ,Can you provide a solution?

Jul 14 17:57:20 node2 kubelet: W0714 17:57:20.540849 17504 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 14 17:57:20 node2 kubelet: E0714 17:57:20.541001 17504 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 14 17:57:23 node2 kubelet: I0714 17:57:23.032330 17504 kubelet.go:1820] skipping pod synchronization - [Failed to start ContainerManager systemd version does not support ability to start a slice as transient unit]

@PLoic

This comment has been minimized.

Copy link
Author

PLoic commented Jul 17, 2017

Sorry for the delay, I was using Weave, I will try to update kubernetes to 1.7.1 and with the new version of weave

@PLoic

This comment has been minimized.

Copy link
Author

PLoic commented Jul 17, 2017

I updated all my components and it seems to works ! :)

@cmluciano

This comment has been minimized.

Copy link
Member

cmluciano commented Jul 17, 2017

Is it ok to close this issue @PLoic ?

@PLoic

This comment has been minimized.

Copy link
Author

PLoic commented Jul 17, 2017

@cmluciano Yes I think it's ok to close this issue

@PLoic PLoic closed this Jul 17, 2017

@skitoo

This comment has been minimized.

Copy link

skitoo commented Jul 18, 2017

Removing the $KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf works for me.
Thanks @PLoic

@bboreham

This comment has been minimized.

Copy link
Contributor

bboreham commented Aug 9, 2017

Note that KUBELET_NETWORK_ARGS is what tells kubelet which kind of network plugin to expect. If you remove it then kubelet expects no plugin, and therefore you get whatever the underlying container runtime gives you: typically Docker "bridge" networking.

This is fine in some cases, particularly if you only have one machine. It is not helpful if you actually want to use CNI.

@ashish-billore

This comment has been minimized.

Copy link

ashish-billore commented Aug 31, 2017

I am seeing the exact same error with kubeadm, where it is struck forever at:

[apiclient] Created API client, waiting for the control plane to become ready

In the "journalctl -r -u kubelet" I see these lines over and over:
Aug 31 16:34:41 k8smaster1 kubelet[8876]: E0831 16:34:41.499982 8876 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 31 16:34:41 k8smaster1 kubelet[8876]: W0831 16:34:41.499746 8876 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d

Version details are:
`kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Kubectl version: Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}`

OS Details are:
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel: Linux 3.10.0-514.el7.x86_64 Architecture: x86-64
Any help is very much appreciated!

@bboreham

This comment has been minimized.

Copy link
Contributor

bboreham commented Aug 31, 2017

@ashish-billore what CNI provider did you install?

@neolit123

This comment has been minimized.

Copy link
Member

neolit123 commented Sep 19, 2017

i'm getting Unable to update cni config: No networks found in /etc/cni/net.d with a recent github tip of the master branch - "v1.9.0-alpha.0.690+9aef242a4c1e42-dirty"

on ubuntu 17.04

if i remove this line from 10-kubelet.conf:
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/ --cni-bin-dir=/opt/cni/bin""

the kubelet starts, then i install weave-net as the pod-network plugin but the kube-system pods never start (they remain scheduled?).

kube-system   etcd-luboitvbox                      0/1       Pending   0          31m
kube-system   kube-apiserver-luboitvbox            0/1       Pending   0          31m
kube-system   kube-controller-manager-luboitvbox   0/1       Pending   0          31m
kube-system   kube-dns-1848271846-7mw9x            0/3       Pending   0          32m
kube-system   kube-proxy-k89jp                     0/1       Pending   0          32m
kube-system   kube-scheduler-luboitvbox            0/1       Pending   0          31m
kube-system   weave-net-v8888                      0/2       Pending   0          30m

same happens with flannel.

@farooqsap

This comment has been minimized.

Copy link

farooqsap commented Mar 28, 2018

Thanks setenforce 0 worked for me.

@Ghostwritten

This comment has been minimized.

Copy link

Ghostwritten commented May 21, 2018

1.$KUBELET_NETWORK_ARGS take out,not solve the problem
2.setenforce 0
3.systemctl stop firewalld
4.docker cgroups drivers:systemd 与 Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
accordance
but,you know the promblem still exist.


May 20 20:10:45 k8s kubelet: I0520 20:10:45.244383 17638 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
May 20 20:10:45 k8s kubelet: E0520 20:10:45.920981 17638 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.18.90:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s.master.com&limit=500&resourceVersion=0: dial tcp 192.168.18.90:6443: getsockopt: connection refused
May 20 20:10:45 k8s kubelet: E0520 20:10:45.924021 17638 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.18.90:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.18.90:6443: getsockopt: connection refused
May 20 20:10:45 k8s kubelet: E0520 20:10:45.935594 17638 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.18.90:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s.master.com&limit=500&resourceVersion=0: dial tcp 192.168.18.90:6443: getsockopt: connection refused

@372046933

This comment has been minimized.

Copy link

372046933 commented May 23, 2018

May 23 10:19:45 arch kubelet[13585]: E0523 10:19:45.909458 13585 kubelet.go:2095] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 23 10:19:46 arch kubelet[13585]: E0523 10:19:46.002646 13585 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 8; ignoring extra CPUs

`sudo kubectl get node

NAME STATUS ROLES AGE VERSION
127.0.0.1 NotReady 23h v1.8.13`

@t3hmrman

This comment has been minimized.

Copy link

t3hmrman commented May 24, 2018

Just ran into this, and it seems to be due to the file actually being empty, here's the output from the install-cni container:

$ k logs canal-25rct install-cni -n kube-system
ls: /calico-secrets: No such file or directory
Wrote Calico CNI binaries to /host/opt/cni/bin
CNI plugin version: v3.1.2
/host/secondary-bin-dir is non-writeable, skipping
CNI config: {
Created CNI config 10-calico.conflist
Done configuring CNI.  Sleep=true

And in /etc/cni/net.d/10-calico.conflist:

$ cat /etc/cni/net.d/10-calico.conflist 
{

When I try to shell into the container (maybe it should be an initContainer?), I get the following:

$ k exec -it canal-25rct -c install-cni -n kube-system -- /bin/bash
Error: Malformed environment entry: "  "name": "k8s-pod-network",
": Success
command terminated with exit code 45

It's weird, because the version of the script hasn't changed, and the only thing I've changed recently is switching to rkt for running containers. Also, this is on Container Linux (CoreOS) if that helps at all.

@waldauf

This comment has been minimized.

Copy link

waldauf commented Jun 1, 2018

Hello K8s folks!

I had many times the same problem. For example - something went wrong during my K8s initialization and I had to use kubeadm reset and initialize K8s again. After run initialization command I got in kubelet log this error:

Jun 01 10:13:40 vncub0626 kubelet[18861]: I0601 10:13:40.665823   18861 kubelet.go:2102] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jun 01 10:13:40 vncub0626 kubelet[18861]: E0601 10:13:40.665874   18861 kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

... I was mad from this error message - nothing helped. So I said myself - at first the initialization run but the re-initialization didn't. So it wasn't caused by this line in Kubelet configuration: KUBELET_NETWORK_ARGS and I don't agree with comment it. So I read kubelet log again and again... and finally I noticed in log next error message:

Jun 01 10:13:29 vncub0626 kubelet[18861]: E0601 10:13:29.376339   18861 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.96.22.11:6443/api/v1/services?limit=500&resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

This error was caused by bad ~/.kube/config file in home directory after previous initialization. After removing it I run initialization again... and voilá... initialization finished successfully. :]

... hope it helps to someone else because this error is nightmare and it's not almost possible to determine its cause.

@kayhu

This comment has been minimized.

Copy link

kayhu commented Jun 7, 2018

@waldauf you are correct!!! awesome!!!

@ChinaSilence

This comment has been minimized.

Copy link

ChinaSilence commented Jun 9, 2018

run follow command works well

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

kubeadm version v1.10.3

@RakeshNagarajan

This comment has been minimized.

Copy link

RakeshNagarajan commented Jun 22, 2018

Removing the $KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf works for me.

@bboreham

This comment has been minimized.

Copy link
Contributor

bboreham commented Jun 22, 2018

Repeating for visibility:

Note that KUBELET_NETWORK_ARGS is what tells kubelet which kind of network plugin to expect. If you remove it then kubelet expects no plugin, and therefore you get whatever the underlying container runtime gives you: typically Docker "bridge" networking.

This is fine in some cases, particularly if you only have one machine. It is not helpful if you actually want to use CNI.

@xxf09th

This comment has been minimized.

Copy link

xxf09th commented Aug 14, 2018

@waldauf thx, it works

@jepsenwan

This comment has been minimized.

Copy link

jepsenwan commented Aug 17, 2018

This came across me when my flannel plugin has not been installed correctly.

@nagarciah

This comment has been minimized.

Copy link

nagarciah commented Aug 22, 2018

I was following this guide today (https://www.techrepublic.com/article/how-to-install-a-kubernetes-cluster-on-centos-7/) and I missed the cgroupfs configuration in "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf". Once I fixed it worked like a charm

@soodakshay

This comment has been minimized.

Copy link

soodakshay commented Sep 3, 2018

#48798 (comment)

@ChinaSilence Can you explain why we have to use flannel? Can't we do it without flannel?

@bilalcaliskan

This comment has been minimized.

Copy link

bilalcaliskan commented Sep 30, 2018

flannel 0.10.0 and kubernetes 1.12.0 can not work together somehow. there is something wrong with kubernetes 1.12.0, so i've downgraded kubernetes to the 1.11.3 and everything is working fine.

Hope kubernetes fixes that issue soon.

@neolit123

This comment has been minimized.

Copy link
Member

neolit123 commented Sep 30, 2018

@bilalx20 i can confirm that - flannel is broken for me too in 1.12.
what you can do is try weave or callico, they work.

@ndlrx

This comment has been minimized.

Copy link

ndlrx commented Sep 30, 2018

I've this issue on my Ubuntu 16.04 with k8s 1.12.

Downgrade to the 1.11.0 and everything is up and running.

@shettyh

This comment has been minimized.

Copy link

shettyh commented Oct 4, 2018

I have this issue on CentOS 7.5 with k8s 1.12

Removing the cni plugin conf from /var/lib/kubelet/kubeadm-flags.env works fine

@ReSearchITEng

This comment has been minimized.

Copy link

ReSearchITEng commented Oct 5, 2018

flanneld needs a fix for k8s 1.12.
Use this PR (till will be approved):
kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
it's a known issue: coreos/flannel#1044

@1e100

This comment has been minimized.

Copy link

1e100 commented Oct 6, 2018

I have experienced the problem @ReSearchITEng describes with 1.12.1. His/her solution worked for me.
EDIT: scratch that, one of the nodes still shows the same issue after kubeadm join
EDIT2: unrelated issue, turns out nvidia-container-runtime was missing on this GPU node. Diagnosed using journalctl -xeu kubelet on the bad node

TL;DR: the solution works

shrmrf added a commit to shrmrf/k8s-resources that referenced this issue Oct 22, 2018

@lanoxx

This comment has been minimized.

Copy link

lanoxx commented Nov 5, 2018

Confirming that the solution from @ReSearchITEng works, I not have my master node in the running state and flannel is up.

@barnabasJ

This comment has been minimized.

Copy link

barnabasJ commented Nov 6, 2018

Just in case somebody is googling this. I had the same problem, in my case the cloud-init process set the NM_CONTROLLED option in /etc/sysconfig/network-scripts/ifcfg-{interface-name} to no.
This option has to be set to yes for NetworkManager to create the needed resolv.conf file for the sdn pods.

@stigok

This comment has been minimized.

Copy link

stigok commented Nov 21, 2018

I had failed to actually apply the weave manifests, hence, no network plugin was initialized.
Note to self: Read the whole Weave installation manual before looking for answers elsewhere 👍

@daohu527

This comment has been minimized.

Copy link

daohu527 commented Jan 8, 2019

flanneld needs a fix for k8s 1.12.
Use this PR (till will be approved):
kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
it's a known issue: coreos/flannel#1044

Additional:
I think this problem cause by kuberadm first init coredns but not init flannel,so it throw "network plugin is not ready: cni config uninitialized".
Solution:

  1. Install flannel by kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
  2. Reset the coredns pod
    kubectl delete coredns-xx-xx
  3. Then run kubectl get pods to see if it works.

if you see this error "cni0" already has an IP address different from 10.244.1.1/24".
follow this:

ifconfig  cni0 down
brctl delbr cni0
ip link delete flannel.1

if you see this error "Back-off restarting failed container", and you can get the log by

root@master:/home/moonx/yaml# kubectl logs coredns-86c58d9df4-x6m9w -n=kube-system
.:53
2019-01-22T08:19:38.255Z [INFO] CoreDNS-1.2.6
2019-01-22T08:19:38.255Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [FATAL] plugin/loop: Forwarding loop detected in "." zone. Exiting. See https://coredns.io/plugins/loop#troubleshooting. Probe query: "HINFO 1599094102175870692.6819166615156126341.".

Then you can see the file "/etc/resolv.conf" on the failed node, if the nameserver is localhost there will be a loopback.Change to:

#nameserver 127.0.1.1
nameserver 8.8.8.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment