Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network not ready after `kubectl apply -f kube-flannel.yaml` in v1.16 cluster #1178

Closed
sshukun opened this issue Sep 19, 2019 · 25 comments
Closed

Comments

@sshukun
Copy link

@sshukun sshukun commented Sep 19, 2019

Expected Behavior

Container network should get ready after kubectl apply -f kube-flannel.yaml

Current Behavior

After creating a v1.16 cluster by kubeadm init and installing flannel cni by kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/32a765fd19ba45b387fdc5e3812c41fff47cfd55/Documentation/kube-flannel.yml, the network still does not get ready and coredns remain pending.

Possible Solution

Steps to Reproduce (for bugs)

  1. Create a v1.16 cluster with kubeadm
  2. kubectl apply -f kube-flannel.yaml

Context

Your Environment

  • Flannel version: v0.11.0
  • Backend used (e.g. vxlan or udp):
  • Etcd version:
  • Kubernetes version (if used): v1.16.0
  • Operating System and version: ubuntu 18.04
  • Link to your project (optional):
@foofoo-2

This comment has been minimized.

Copy link

@foofoo-2 foofoo-2 commented Sep 19, 2019

Same behaviour here, the only difference is the underlying OS:
CentOS Linux release 7.6.1810 (Core)

@lff0305

This comment has been minimized.

Copy link

@lff0305 lff0305 commented Sep 19, 2019

Same behavior here

On CentOS 7 Linux node1 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
[root@node1 ~]#

journalctl -f -u kubelet.service :

Sep 19 04:21:44 node1 kubelet[10636]: W0919 04:21:44.389281   10636 cni.go:202] Error validating CNI config &{cbr0  false [0xc001613540 0xc0016135c0] [123 10 32 32 34 110 97 109 101 34 58 32 34 99 98 114 48 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 102 108 97 110 110 101 108 34 44 10 32 32 32 32 32 32 34 100 101 108 101 103 97 116 101 34 58 32 123 10 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 10 32 32 93 10 125 10]}: [plugin flannel does not support config version ""]
Sep 19 04:21:44 node1 kubelet[10636]: W0919 04:21:44.389341   10636 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 19 04:21:45 node1 kubelet[10636]: E0919 04:21:45.583516   10636 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

There was message [plugin flannel does not support config version ""

@drkhosla

This comment has been minimized.

Copy link

@drkhosla drkhosla commented Sep 19, 2019

I also encounter same error

Linux hrshl 5.3.0-1.el7.elrepo.x86_64 #1 SMP Mon Sep 16 08:44:46 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@hrsh ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY   STATUS              RESTARTS   AGE
kube-system   coredns-5644d7b6d9-hq7qp          0/1     ContainerCreating   0          96m
kube-system   coredns-5644d7b6d9-wq2c7          0/1     ContainerCreating   0          96m
kube-system   etcd-hrsh                      1/1     Running             4          95m
kube-system   kube-apiserver-hrsh            1/1     Running             4          96m
kube-system   kube-controller-manager-hrsh   1/1     Running             5          96m
kube-system   kube-flannel-ds-amd64-p4nwk       1/1     Running             1          15m
kube-system   kube-proxy-pfh9d                  1/1     Running             4          96m
kube-system   kube-scheduler-hrsh            1/1     Running             5          96m

[root@hrsh ~]# kubectl describe pod coredns-5644d7b6d9-hq7qp -n kube-system
Name:                 coredns-5644d7b6d9-hq7qp
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hrsh/xxx.xxx.xxx.84
Start Time:           Thu, 19 Sep 2019 16:32:38 +0530
Labels:               k8s-app=kube-dns
                      pod-template-hash=5644d7b6d9
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-5644d7b6d9
Containers:
  coredns:
    Container ID:  
    Image:         k8s.gcr.io/coredns:1.6.2
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-8hbwc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-8hbwc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-8hbwc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason           Age                    From              Message
  ----     ------           ----                   ----              -------
  Warning  NetworkNotReady  20m (x1955 over 85m)   kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  NetworkNotReady  6m59s (x307 over 17m)  kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  NetworkNotReady  60s (x25 over 109s)    kubelet, hrsh  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[root@hrsh ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

[root@hrsh ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2019-09-19 18:05:59 IST; 5min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 1009 (kubelet)
    Tasks: 22
   Memory: 124.5M
   CGroup: /system.slice/kubelet.service
           └─1009 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

Sep 19 18:11:05 hrsh kubelet[1009]: E0919 18:11:05.452153    1009 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:06 hrsh kubelet[1009]: E0919 18:11:06.096629    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:06 hrsh kubelet[1009]: E0919 18:11:06.096870    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: E0919 18:11:08.096630    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: E0919 18:11:08.096904    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:08 hrsh kubelet[1009]: W0919 18:11:08.850500    1009 cni.go:202] Error validating CNI config &{cbr0  false [0xc001701fe0 0xc00171c060] [123 10 32 32 34 110 97 109 101 34 58 32 34 99 98 114 48 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 102 108 97 110 110 101 108 34 44 10 32 32 32 32 32 32 34 100 101 108 101 103 97 116 101 34 58 32 123 10 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 10 32 32 93 10 125 10]}: [plugin flannel does not support config version ""]
Sep 19 18:11:08 hrsh kubelet[1009]: W0919 18:11:08.850567    1009 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.096604    1009 pod_workers.go:191] Error syncing pod f2ca516a-166f-48da-b54b-6cc8d6750469 ("coredns-5644d7b6d9-hq7qp_kube-system(f2ca516a-166f-48da-b54b-6cc8d6750469)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.096765    1009 pod_workers.go:191] Error syncing pod 1d7bbe8c-3736-43e1-93f6-bf3c043195ab ("coredns-5644d7b6d9-wq2c7_kube-system(1d7bbe8c-3736-43e1-93f6-bf3c043195ab)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Sep 19 18:11:10 hrsh kubelet[1009]: E0919 18:11:10.474639    1009 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

[root@hrsh ~]# docker image list
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver            v1.16.0             b305571ca60a        22 hours ago        217MB
k8s.gcr.io/kube-controller-manager   v1.16.0             06a629a7e51c        22 hours ago        163MB
k8s.gcr.io/kube-proxy                v1.16.0             c21b0c7400f9        22 hours ago        86.1MB
k8s.gcr.io/kube-scheduler            v1.16.0             301ddc62b80b        22 hours ago        87.3MB
k8s.gcr.io/etcd                      3.3.15-0            b2756210eeab        2 weeks ago         247MB
k8s.gcr.io/coredns                   1.6.2               bf261d157914        5 weeks ago         44.1MB
quay.io/coreos/flannel               v0.11.0-amd64       ff281650a721        7 months ago        52.6MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        21 months ago       742kB

@tobiasvdp

This comment has been minimized.

Copy link

@tobiasvdp tobiasvdp commented Sep 19, 2019

same here ubuntu 1.18, docker version 18.06.3

@smsimeonov

This comment has been minimized.

Copy link

@smsimeonov smsimeonov commented Sep 19, 2019

Same behaviour here with Debian Buster (10.1)
Commenting for updates

@sirdarkat

This comment has been minimized.

Copy link

@sirdarkat sirdarkat commented Sep 19, 2019

Same here as tobiasvdp. Latest version of Flannel and nodes are telling me Not Ready the container network is not working. This occurred after updating to version 1.16 of kubernetes. Luckily it's dev so we can just hold off until a fix is applied.

@schizofreindly

This comment has been minimized.

Copy link

@schizofreindly schizofreindly commented Sep 19, 2019

Same here with Ubuntu 18.04. Also node selector namespace is beta.kubernetes.io which was deprecated and now kubernetes.io. I replaced it in yaml but still no network.

@dougbtv

This comment has been minimized.

Copy link

@dougbtv dougbtv commented Sep 19, 2019

I was able to get nodes to come up as ready by adding:

      "cniVersion":"0.3.1",

Under this line:

"name": "cbr0",

When I do a kubectl create -f the-above-flannel.yml

@schizofreindly

This comment has been minimized.

Copy link

@schizofreindly schizofreindly commented Sep 19, 2019

I was able to get nodes to come up as ready by adding:

      "cniVersion":"0.3.1",

Under this line:

"name": "cbr0",

When I do a kubectl create -f the-above-flannel.yml

Works for me. How did you know ?

@jar3b

This comment has been minimized.

Copy link

@jar3b jar3b commented Sep 19, 2019

@dougbtv, works for me to.

@sirdarkat

This comment has been minimized.

Copy link

@sirdarkat sirdarkat commented Sep 20, 2019

@dougbtv Awesome find, this worked for me as well.

If you can give a response on why this works I would love to know why. Thanks

@sshukun

This comment has been minimized.

Copy link
Author

@sshukun sshukun commented Sep 20, 2019

It seems like in v1.16, kubelet will validate cni config file and if the cniVersion is not supported, an error will be returned.
kubernetes/kubernetes#80482
Currently cni-conf.json data in the configmap of kube-flannel.yaml lacks the cniVersion field, so an error like plugin flannel does not support config version "" is returned.

Seems we need to add supported cniVersion to kube-flannel.yaml.

@sshukun

This comment has been minimized.

Copy link
Author

@sshukun sshukun commented Sep 20, 2019

#1181 helps with this

@rayyen

This comment has been minimized.

Copy link

@rayyen rayyen commented Sep 20, 2019

#1181 helps with this

this does work on my bare-metal clusters with 1.16.0.
have to insert "cniVersion": "0.2.0" to /etc/cni/net.d/10-flannel.conflist on every node !!

@drkhosla

This comment has been minimized.

Copy link

@drkhosla drkhosla commented Sep 20, 2019

Thanks @dougbtv, works for me to.

@weiwunb

This comment has been minimized.

Copy link

@weiwunb weiwunb commented Sep 21, 2019

Got the problem that can not find cp command in the PATH
Host OS is Ubuntu 18.04

@nbroeking

This comment has been minimized.

Copy link

@nbroeking nbroeking commented Sep 22, 2019

Same here on a raspberry pi 4 - arm 64

@AHassanSOS

This comment has been minimized.

Copy link

@AHassanSOS AHassanSOS commented Sep 26, 2019

This created some iptables rules that prevented ssh from working.

@sonishi85

This comment has been minimized.

Copy link

@sonishi85 sonishi85 commented Sep 26, 2019

@schizofreindly - pure gold. thank you. wasted a day trying all sorts of things before I stumbled across this post.

@oomichi

This comment has been minimized.

Copy link

@oomichi oomichi commented Sep 26, 2019

Thanks @schizofreindly That works for me.

@arasananbu

This comment has been minimized.

Copy link
Contributor

@arasananbu arasananbu commented Sep 27, 2019

Fixed by #1174

@rajatchopra

This comment has been minimized.

Copy link
Contributor

@rajatchopra rajatchopra commented Sep 27, 2019

Closing as per above comment.

@isnuryusuf

This comment has been minimized.

Copy link

@isnuryusuf isnuryusuf commented Oct 2, 2019

work for me,

CentOS Linux release 7.7.1908 (Core)

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

@yonahd

This comment has been minimized.

Copy link

@yonahd yonahd commented Nov 11, 2019

This is really helpful. I think flannel should come out with a revision or put it in the docs under kubernetes 1.16 support in order for users to notice this issue.
“Lessons learned from upgrading kubernetes offline to 1.16” https://link.medium.com/j6FVuVBxv1

@rushikeshjadhav

This comment has been minimized.

Copy link

@rushikeshjadhav rushikeshjadhav commented Nov 19, 2019

#1181 helps with this

this does work on my bare-metal clusters with 1.16.0.
have to insert "cniVersion": "0.2.0" to /etc/cni/net.d/10-flannel.conflist on every node !!

Added "cniVersion":"0.3.1" to /etc/cni/net.d/10-flannel.conflist and got it working

{
  "name": "cbr0",
  "cniVersion":"0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.