network plugin is not ready: cni config uninitialized

[PLoic]

Hello, I want to do a fresh install of kubernetes via kubeadm, but when I start the install I'm stuck on

[apiclient] Created API client, waiting for the control plane to become ready

When I do a journalctl -xe I see :

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

And I don't why I get this error. I also tried to disable firewalld but no effect.

Environment:

• Kubernetes version (use kubectl version): v1.7.0
• Cloud provider or hardware configuration**:
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): 3.10.0-514.26.2.el7.x86_64
• Install tools: Kubeadm
• Others:
  docker version : Docker version 17.06.0-ce, build 02c1d87
  My RPM version :

kubeadm-1.7.0, kubectl-1.7.0, kubelet-1.7.0, kubernetes-cni-0.5.1

Thanks for your help

[k8s-github-robot]

@PLoic There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

[PLoic]

/area [kubeadm]

[dcbw]

@PLoic you get this error because no CNI network has been defined in /etc/cni/net.d and you're apparently using the CNI network plugin. Something has to write a config file to that directory to tell the CNI driver how to configure networking. I'm not sure what/how kubeadm does that though, so I'll leave that to @jbeda or other kubeadm folks.

[xiangpengzhao]

xref: #43567

[alioliver]

@dcbw Hi dcbw , Environment same to @PLoic , but I get this same error

[PLoic]

It's seem to working by removing the$KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[tiendungitd]

removing $KUBELET_NETWORK_ARGS not work with me.

[alioliver]

@PLoic also not work with me

[bboreham]

@PLoic at step 3 which pod network did you install? There are various choices, and troubleshooting after that depends on the specific case.

[dcbw]

@PLoic also, kubelet logs would be great

[tiendungitd]

try to apply this plugin: kubectl apply --filename https://git.io/weave-kube-1.6
it works for me.

[ljb-2000]

@PLoic @dcbw 　I install flannel plugin of k8s(1.7) still get this same error ,Can you provide a solution?

Jul 14 17:57:20 node2 kubelet: W0714 17:57:20.540849 17504 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 14 17:57:20 node2 kubelet: E0714 17:57:20.541001 17504 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 14 17:57:23 node2 kubelet: I0714 17:57:23.032330 17504 kubelet.go:1820] skipping pod synchronization - [Failed to start ContainerManager systemd version does not support ability to start a slice as transient unit]

[PLoic]

Sorry for the delay, I was using Weave, I will try to update kubernetes to 1.7.1 and with the new version of weave

[PLoic]

I updated all my components and it seems to works ! :)

[cmluciano]

Is it ok to close this issue @PLoic ?

[PLoic]

@cmluciano Yes I think it's ok to close this issue

[skitoo]

Removing the $KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf works for me.
Thanks @PLoic

[bboreham]

Note that KUBELET_NETWORK_ARGS is what tells kubelet which kind of network plugin to expect. If you remove it then kubelet expects no plugin, and therefore you get whatever the underlying container runtime gives you: typically Docker "bridge" networking.

This is fine in some cases, particularly if you only have one machine. It is not helpful if you actually want to use CNI.

[ashish-billore]

I am seeing the exact same error with kubeadm, where it is struck forever at:

[apiclient] Created API client, waiting for the control plane to become ready

In the "journalctl -r -u kubelet" I see these lines over and over:
Aug 31 16:34:41 k8smaster1 kubelet[8876]: E0831 16:34:41.499982 8876 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 31 16:34:41 k8smaster1 kubelet[8876]: W0831 16:34:41.499746 8876 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d

Version details are:
`kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Kubectl version: Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}`

OS Details are:
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel: Linux 3.10.0-514.el7.x86_64 Architecture: x86-64
Any help is very much appreciated!

[bboreham]

@ashish-billore what CNI provider did you install?

[neolit123]

i'm getting Unable to update cni config: No networks found in /etc/cni/net.d with a recent github tip of the master branch - "v1.9.0-alpha.0.690+9aef242a4c1e42-dirty"

on ubuntu 17.04

if i remove this line from 10-kubelet.conf:
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/ --cni-bin-dir=/opt/cni/bin""

the kubelet starts, then i install weave-net as the pod-network plugin but the kube-system pods never start (they remain scheduled?).

kube-system   etcd-luboitvbox                      0/1       Pending   0          31m
kube-system   kube-apiserver-luboitvbox            0/1       Pending   0          31m
kube-system   kube-controller-manager-luboitvbox   0/1       Pending   0          31m
kube-system   kube-dns-1848271846-7mw9x            0/3       Pending   0          32m
kube-system   kube-proxy-k89jp                     0/1       Pending   0          32m
kube-system   kube-scheduler-luboitvbox            0/1       Pending   0          31m
kube-system   weave-net-v8888                      0/2       Pending   0          30m

same happens with flannel.

[RakeshNagarajan]

@mdzddl you deleted --network-plugin=cni cause kubelet complains about cni? Not so clever. Deleting the default network plugin is not recommended at all.

Then what is the solution

[sampleref]

flanneld needs a fix for k8s 1.12.
Use this PR (till will be approved):
kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
it's a known issue: coreos/flannel#1044

Additional:
I think this problem cause by kuberadm first init coredns but not init flannel,so it throw "network plugin is not ready: cni config uninitialized".
Solution:

1. Install flannel by kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
2. Reset the coredns pod
   kubectl delete coredns-xx-xx
3. Then run kubectl get pods to see if it works.

if you see this error "cni0" already has an IP address different from 10.244.1.1/24".
follow this:

ifconfig  cni0 down
brctl delbr cni0
ip link delete flannel.1

if you see this error "Back-off restarting failed container", and you can get the log by

root@master:/home/moonx/yaml# kubectl logs coredns-86c58d9df4-x6m9w -n=kube-system
.:53
2019-01-22T08:19:38.255Z [INFO] CoreDNS-1.2.6
2019-01-22T08:19:38.255Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [FATAL] plugin/loop: Forwarding loop detected in "." zone. Exiting. See https://coredns.io/plugins/loop#troubleshooting. Probe query: "HINFO 1599094102175870692.6819166615156126341.".

Then you can see the file "/etc/resolv.conf" on the failed node, if the nameserver is localhost there will be a loopback.Change to:

#nameserver 127.0.1.1
nameserver 8.8.8.8

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"

Throws error as above

[neolit123]

flannel has not updated their manifest to comply with the latest changes in k8s 1.16.
try a different CNI plugin, like Calico or WeaveNet.

[neolit123]

...or patch the flannel manifests to use apps/v1 instead of extensions/v1beta1

[juxuny]

I have this issue on Ubuntu 16.04 with k8s 1.16 (I run ubuntu on vagrant)

Removing the cni plugin conf from /var/lib/kubelet/kubeadm-flags.env works fine

[pohly]

flannel has not updated their manifest to comply with the latest changes in k8s 1.16.
try a different CNI plugin, like Calico or WeaveNet.
...
...or patch the flannel manifests to use apps/v1 instead of extensions/v1beta1

That was fixed a while ago, but the links in the Kubernetes documentation still point to an older version which doesn't work (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ has https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml). Using "master" instead works and also fixes another issue (missing version in the CNI config).

[neolit123]

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

uses https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

which should include both the cniVersion and apps/v1 fixes.

[coderdba]

This is what I saw: This error comes when you dont have flannel running yet, but you start kublet with just manifests of apiserver, scheduler and controller-manager - WHILE YOU HAVE THIS LINE in 10-kubeadm.conf - "Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --node-ip=192.168.8.11"

Comment that and start kubelet.
Then core kube-system pods come up.
Then install kube-proxy
Then install flannel
*** Then uncomment the above line and restart kubelet ***
Install core-dns/kube-dns

[mvaknine]

Hi,
i am trying to install version 1.16.0 i am using kuberouter pluging and i am getting the same errors

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[juxuny]

Hi,
i am trying to install version 1.16.0 i am using kuberouter pluging and i am getting the same errors

Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

If you can provide more information about the running environment, it will be helpful. Such as, the Operate System, and what did you do before the error occurred.

[mvaknine]

this is a new fresh install of version 1.16.0 on amazon.
i am usuing this AMI - k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
uname -a
Linux ip-172-28-125-218 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

if i install 1.15.0 there are no problems at all.

this is what i see in syslog of the master nodes.

Mar 12 05:26:22 ip-172-28-125-218 kubeletÄ3656Å: E0312 05:26:22.761009 3656 kubelet.go:2187Å Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Mar 12 05:26:25 ip-172-28-125-218 dockerÄ3570Å: I0312 05:26:25.713681 3619 dns.go:47Å DNSView unchanged: 5
Mar 12 05:26:25 ip-172-28-125-218 kubeletÄ3656Å: W0312 05:26:25.883857 3656 cni.go:202Å Error validating CNI config &äkubernetes false Ä0xc0009d8260Å Ä123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 34 44 34 110 97 109 101 34 58 34 107 117 98 101 114 110 101 116 101 115 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 98 114 105 100 103 101 34 58 34 107 117 98 101 45 98 114 105 100 103 101 34 44 34 105 112 97 109 34 58 123 34 115 117 98 110 101 116 34 58 34 49 48 48 46 57 54 46 48 46 48 47 50 52 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 108 111 99 97 108 34 125 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 107 117 98 101 114 110 101 116 101 115 34 44 34 116 121 112 101 34 58 34 98 114 105 100 103 101 34 125 93 125Åå: Äplugin bridge does not support config version ""Å
Mar 12 05:26:25 ip-172-28-125-218 kubeletÄ3656Å: W0312 05:26:25.883925 3656 cni.go:237Å Unable to update cni config: no valid networks found in /etc/cni/net.d/
Mar 12 05:26:27 ip-172-28-125-218 kubeletÄ3656Å: E0312 05:26:27.762309 3656 kubelet.go:2187Å Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Mar 12 05:26:30 ip-172-28-125-218 dockerÄ3570Å: I0312 05:26:30.713906 3619 dns.go:47Å DNSView unchanged: 5
Mar 12 05:26:30 ip-172-28-125-218 kubeletÄ3656Å: W0312 05:26:30.886362 3656 cni.go:202Å Error validating CNI config &äkubernetes false Ä0xc0008fc000Å Ä123 34 99 110 105 86 101 114 115 105 111 110 34 58 34 34 44 34 110 97 109 101 34 58 34 107 117 98 101 114 110 101 116 101 115 34 44 34 112 108 117 103 105 110 115 34 58 91 123 34 98 114 105 100 103 101 34 58 34 107 117 98 101 45 98 114 105 100 103 101 34 44 34 105 112 97 109 34 58 123 34 115 117 98 110 101 116 34 58 34 49 48 48 46 57 54 46 48 46 48 47 50 52 34 44 34 116 121 112 101 34 58 34 104 111 115 116 45 108 111 99 97 108 34 125 44 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 107 117 98 101 114 110 101 116 101 115 34 44 34 116 121 112 101 34 58 34 98 114 105 100 103 101 34 125 93 125Åå: Äplugin bridge does not support config version ""Å
Mar 12 05:26:30 ip-172-28-125-218 kubeletÄ3656Å: W0312 05:26:30.886428 3656 cni.go:237Å Unable to update cni config: no valid networks found in /etc/cni/net.d/

[hien]

Removing the cni plugin conf from /var/lib/kubelet/kubeadm-flags.env also works on CentOS 7.6 with k8s 1.16.8

[ikramuallah]

please don't change anything. Just run this command. The error will be gone.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

[ganesh-k13]

Thanks @ikramuallah , that worked for 1.18, thing is it didn't work directly cause one of the flannel pod could not be pulled as one of the quay sites was throwing 500. So my suggestion is that after applying the YAML, check if all the flannel pods have come up and debug that. Linking issue I raised in flannel for reference. flannel-io/flannel#1294

[yoshieki1992]

please don't change anything. Just run this command. The error will be gone.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

thanks!

[juxuny]

please don't change anything. Just run this command. The error will be gone.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

thanks!

Sorry, I can't reach the link. Is something wrong ?

[ReSearchITEng]

@juxuny link is ok, maybe some temporary netw connectivity issues?

[snehap25]

Please someone help me
For kube-proxy-windows-xhxzw pod status is ContainerCreating .When I describe the pods it gives warning as

Warning NetworkNotReady 3m27s (x4964 over 168m) kubelet, casts1 network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

All others pods are in running state. except the kube-proxy-windows

I have created kubernetes envrioronment as below:
Master Node Ready V1.19.0 RHEL 7 OS
Worker Node NotReady V1.19.0 Windows Server 2019
I am trying to join Windows worker node to master node

I am using flannel network. I have tried below solutions:
1.kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
2. Change 10-flannel.confglist's cniVersion from "0.3.1" to "0.2.0"
10-flannel.confglist was already there.

Please let me know the exact problem

[vetal2504]

In my case when i am init kubernetes master i have had this issue. After delete all data in etcd the process init was successful
On all etcd nodes:
systemctl stop etcd
rm -rf /var/lib/etcd/*
systemctl daemon-reload && systemctl enable etcd && systemctl start

[echenim]

I had same problem of node status NotReady with my kubernetes node, below is what worked for me

OS Version: Ubuntu Server 20.10
I commented out #KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2"
Removing from /var/lib/kubelet/kubeadm-flags.env works fine

[v0ldemort]

I am facing the same error, not getting any clue to how to rid of this - please help

Error - runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Node has been created but not adding to cluster,

[ZW007]

It's seem to working by removing the$KUBELET_NETWORK_ARGS in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

@PLoic I am using Ubuntu 18.04, my /etc/systemd/system/kubelet.service.d/10-kubeadm.conf has no $KUBELET_NETWORK_ARGS What should I do (

[prbakhsh]

Removing the cni plugin conf from /var/lib/kubelet/kubeadm-flags.env also works on CentOS 7.6 with k8s 1.16.8

its works for me on oracle linux 7.7
Thanks indeed

[aahemm]

Hey folks!
I had the same problem. I use containerd as runtime. I was using http_proxy environment variable for containerd service and that was the problem. So I excluded cluster IP ranges (10.0.0.0/24 and 192.168.0.0/24) in no_proxy and the problem was gone.
Hope it helps

[tangducthinh456]

I'm using VMWare to create cluster Kubernetes and get same error, after shutdown and restart VM it works!

[ScGPS]

All nodes install:

sudo mkdir -p /opt/cni/bin
cd /opt/cni/bin
sudo curl -L --insecure -O https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
sudo gunzip cni-plugins-linux-amd64-v0.8.6.tgz
sudo tar -xvf cni-plugins-linux-amd64-v0.8.6.tar

[xlw712]

All nodes install:

sudo mkdir -p /opt/cni/bin
cd /opt/cni/bin
sudo curl -L --insecure -O https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
sudo gunzip cni-plugins-linux-amd64-v0.8.6.tgz
sudo tar -xvf cni-plugins-linux-amd64-v0.8.6.tar

I'm copy greate node /opt/cni/bin/flannel to bad node folder ,but node status is NotReady too, exec describe node log below

Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----             ------    -----------------                 ------------------                ------              -------
  MemoryPressure   Unknown   Wed, 22 Mar 2023 20:16:52 +0800   Wed, 22 Mar 2023 20:18:47 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure     Unknown   Wed, 22 Mar 2023 20:16:52 +0800   Wed, 22 Mar 2023 20:18:47 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure      Unknown   Wed, 22 Mar 2023 20:16:52 +0800   Wed, 22 Mar 2023 20:18:47 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready            Unknown   Wed, 22 Mar 2023 20:16:52 +0800   Wed, 22 Mar 2023 20:18:47 +0800   NodeStatusUnknown   Kubelet stopped posting node status.