Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp 127.0.0.1:9099: connect: connection refused #4087

Closed
wybaron opened this issue Oct 13, 2020 · 4 comments
Closed

calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp 127.0.0.1:9099: connect: connection refused #4087

wybaron opened this issue Oct 13, 2020 · 4 comments
Labels
kind/support

Comments

@wybaron
Copy link

wybaron commented Oct 13, 2020

1.Basic environment

1.1 The internet

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp9s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:4b:65:20 brd ff:ff:ff:ff:ff:ff
3: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 68:91:d0:60:cc:3a brd ff:ff:ff:ff:ff:ff
    inet 172.27.9.211/22 brd 172.27.11.255 scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::6a91:d0ff:fe60:cc3a/64 scope link
       valid_lft forever preferred_lft forever
4: enp10s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:4b:65:21 brd ff:ff:ff:ff:ff:ff
5: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 68:91:d0:60:cc:3b brd ff:ff:ff:ff:ff:ff
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:a2:4a:a9:a5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
7: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
...

1.2 DNS

nameserver 8.8.8.8

1.3 /etc/hosts Mapping configuration

# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 k8s01

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
xxx.xx.x.xx1 apiserver.cluster.local
xxx.xx.x.xx1 k8s01
xxx.xx.x.xx2 k8s02
xxx.xx.x.xx3 k8s03
xxx.xx.x.xx4 k8s04
xxx.xx.x.xx5 k8s05

2.Abnormal

2.1 source

# kubectl get pods -n kube-system
NAME                                       READY   STATUS             RESTARTS   AGE
calico-kube-controllers-6b4c6ff56f-5lgpb   1/1     Running            0          14h
calico-node-25jm4                          0/1     CrashLoopBackOff   203        14h
calico-node-2kcfw                          0/1     CrashLoopBackOff   204        14h
calico-node-fwd88                          0/1     Running            213        14h
calico-node-kr24t                          0/1     CrashLoopBackOff   215        14h
calico-node-msqdn                          0/1     CrashLoopBackOff   204        14h
...

2.2 logs

# kubectl describe pod calico-node-25jm4 -n kube-system
...
Events:
  Type     Reason     Age                     From            Message
  ----     ------     ----                    ----            -------
  Warning  Unhealthy  14m (x1952 over 14h)    kubelet, k8s04  Readiness probe failed: calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp 127.0.0.1:9099: connect: connection refused
  Warning  BackOff    3m57s (x2392 over 14h)  kubelet, k8s04  Back-off restarting failed container

3.Ways that have been tried (but still not successfully resolved)

3.1 Regular expression specifies calico network card

# kubectl delete -f calico.yaml
# vim calico.yaml
...
            - name: IP_AUTODETECTION_METHOD
              value: "interface=enp.*|eth.*|en.*"
...
# kubectl apply -f calico.yaml

I configured through the above methods, but still can't solve this problem, I hope to get your help.thank you very much!
@tscswcn
Copy link

tscswcn commented Oct 13, 2020

I think it is readiness issue

@lwr20
Copy link
Member

lwr20 commented Oct 15, 2020

What calico version is this? Where did you get the calico manifest from?

@caseydavenport
Copy link
Member

Readiness won't cause the pods to crashloop. Though the liveness probes would.

I think we need logs from the failing calico/node pods in order to diagnose this. @tscswcn can you provide those?

@tirelibirefe
Copy link

tirelibirefe commented Feb 24, 2022 •
edited
Loading 

LAST SEEN   TYPE      REASON                   OBJECT                                          MESSAGE
3m14s       Normal    Scheduled                pod/calico-kube-controllers-55b67d5544-kncnd    Successfully assigned calico-system/calico-kube-controllers-55b67d5544-kncnd to ip-10-60-66-177.eu-central-1.compute.internal
3m14s       Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5dbe288d04afa894c52d62a317c1a1a1389cb532b16c1505893894fe256526c6": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
3m3s        Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5d9be67f3611a6e2c4982b5de56f5034e072242168da71e63e54bc49c80aa7d": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2m49s       Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "14e0dfa9f207d51d87607b331dfc49c0b4c3ebd4fdf0456edea5bd50fb4ee35c": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2m37s       Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f7e3bbb8db43435d70f53ac1e6de1f73db2edcd51fdfe372ef68e148872b624c": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2m26s       Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "36744c19c1a0fa8bbb3c2a6d22998e8dd641c1ded6584476a1dd852ac289ae68": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2m14s       Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "137c9abd3ec8b7b2d4d0332b54f1865489646b367d2236e82a1aae0f11f4f84f": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
2m1s        Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7a7c4004bbabe480dd0e8d55008a7717b4559864970da895316b657ab012816c": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
106s        Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e276e7eeebd323995f2c31bed240179be59665981f239c9e93a1c4daf4fce162": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
92s         Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ea6baf7d056bf1820b5b0f1d66878e0381e950fad0bf737a6cac59903fbfc0ae": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
4s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "11bfee8532ecf9586f435b29acc86586cf819f0b1a8c0fb96a579c550ba1296e": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
3m14s       Normal    SuccessfulCreate         replicaset/calico-kube-controllers-55b67d5544   Created pod: calico-kube-controllers-55b67d5544-kncnd
3m14s       Normal    ScalingReplicaSet        deployment/calico-kube-controllers              Scaled up replica set calico-kube-controllers-55b67d5544 to 1
3m14s       Normal    Scheduled                pod/calico-node-cp8hh                           Successfully assigned calico-system/calico-node-cp8hh to ip-10-60-66-177.eu-central-1.compute.internal
3m14s       Normal    Pulled                   pod/calico-node-cp8hh                           Container image "docker.io/calico/pod2daemon-flexvol:v3.22.0" already present on machine
3m14s       Normal    Created                  pod/calico-node-cp8hh                           Created container flexvol-driver
3m14s       Normal    Started                  pod/calico-node-cp8hh                           Started container flexvol-driver
3m13s       Normal    Pulled                   pod/calico-node-cp8hh                           Container image "docker.io/calico/node:v3.22.0" already present on machine
3m13s       Normal    Created                  pod/calico-node-cp8hh                           Created container calico-node
3m13s       Normal    Started                  pod/calico-node-cp8hh                           Started container calico-node
3m12s       Warning   Unhealthy                pod/calico-node-cp8hh                           Readiness probe failed: calico/node is not ready: felix is not ready: Get "http://localhost:9099/readiness": dial tcp 127.0.0.1:9099: connect: connection refused
3m14s       Normal    Scheduled                pod/calico-node-f944f                           Successfully assigned calico-system/calico-node-f944f to ip-10-60-20-88.eu-central-1.compute.internal
3m14s       Normal    Pulled                   pod/calico-node-f944f                           Container image "docker.io/calico/pod2daemon-flexvol:v3.22.0" already present on machine
3m14s       Normal    Created                  pod/calico-node-f944f                           Created container flexvol-driver
3m14s       Normal    Started                  pod/calico-node-f944f                           Started container flexvol-driver
3m13s       Normal    Pulled                   pod/calico-node-f944f                           Container image "docker.io/calico/node:v3.22.0" already present on machine
3m13s       Normal    Created                  pod/calico-node-f944f                           Created container calico-node
3m13s       Normal    Started                  pod/calico-node-f944f                           Started container calico-node
3m12s       Warning   Unhealthy                pod/calico-node-f944f                           Readiness probe failed: calico/node is not ready: felix is not ready: Get "http://localhost:9099/readiness": dial tcp 127.0.0.1:9099: connect: connection refused
3m14s       Normal    Scheduled                pod/calico-node-kztsh                           Successfully assigned calico-system/calico-node-kztsh to ip-10-60-53-225.eu-central-1.compute.internal
3m14s       Normal    Pulled                   pod/calico-node-kztsh                           Container image "docker.io/calico/pod2daemon-flexvol:v3.22.0" already present on machine
3m14s       Normal    Created                  pod/calico-node-kztsh                           Created container flexvol-driver
3m14s       Normal    Started                  pod/calico-node-kztsh                           Started container flexvol-driver
3m13s       Normal    Pulled                   pod/calico-node-kztsh                           Container image "docker.io/calico/node:v3.22.0" already present on machine
3m13s       Normal    Created                  pod/calico-node-kztsh                           Created container calico-node
3m13s       Normal    Started                  pod/calico-node-kztsh                           Started container calico-node
3m12s       Warning   Unhealthy                pod/calico-node-kztsh                           Readiness probe failed: calico/node is not ready: felix is not ready: Get "http://localhost:9099/readiness": dial tcp 127.0.0.1:9099: connect: connection refused
3m14s       Normal    SuccessfulCreate         daemonset/calico-node                           Created pod: calico-node-kztsh
3m14s       Normal    SuccessfulCreate         daemonset/calico-node                           Created pod: calico-node-f944f
3m14s       Normal    SuccessfulCreate         daemonset/calico-node                           Created pod: calico-node-cp8hh
3m6s        Normal    Scheduled                pod/calico-typha-6cbdfd7b58-psf4w               Successfully assigned calico-system/calico-typha-6cbdfd7b58-psf4w to ip-10-60-53-225.eu-central-1.compute.internal
3m5s        Normal    Pulling                  pod/calico-typha-6cbdfd7b58-psf4w               Pulling image "docker.io/calico/typha:v3.22.0"
3m1s        Normal    Pulled                   pod/calico-typha-6cbdfd7b58-psf4w               Successfully pulled image "docker.io/calico/typha:v3.22.0" in 4.478576474s
3m          Normal    Created                  pod/calico-typha-6cbdfd7b58-psf4w               Created container calico-typha
3m          Normal    Started                  pod/calico-typha-6cbdfd7b58-psf4w               Started container calico-typha
3m14s       Normal    Scheduled                pod/calico-typha-6cbdfd7b58-xmmqs               Successfully assigned calico-system/calico-typha-6cbdfd7b58-xmmqs to ip-10-60-20-88.eu-central-1.compute.internal
3m14s       Normal    Pulling                  pod/calico-typha-6cbdfd7b58-xmmqs               Pulling image "docker.io/calico/typha:v3.22.0"
3m9s        Normal    Pulled                   pod/calico-typha-6cbdfd7b58-xmmqs               Successfully pulled image "docker.io/calico/typha:v3.22.0" in 4.144873054s
3m9s        Normal    Created                  pod/calico-typha-6cbdfd7b58-xmmqs               Created container calico-typha
3m9s        Normal    Started                  pod/calico-typha-6cbdfd7b58-xmmqs               Started container calico-typha
3m14s       Normal    SuccessfulCreate         replicaset/calico-typha-6cbdfd7b58              Created pod: calico-typha-6cbdfd7b58-xmmqs
3m6s        Normal    SuccessfulCreate         replicaset/calico-typha-6cbdfd7b58              Created pod: calico-typha-6cbdfd7b58-psf4w
3m15s       Normal    NoPods                   poddisruptionbudget/calico-typha                No matching pods found
3m14s       Normal    ScalingReplicaSet        deployment/calico-typha                         Scaled up replica set calico-typha-6cbdfd7b58 to 1
3m6s        Normal    ScalingReplicaSet        deployment/calico-typha                         Scaled up replica set calico-typha-6cbdfd7b58 to 2
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eade90ec78643927bfa771a7da9a2ca7eec55e7d6c080c32456828b868656c6d": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a1306d5cdc7c887ad11be590ac5ff072a339511193c08b3daac6526b7ed8e794": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "20a447d3635dacba3b1b88cefd2c4c20f66ccd7df0a8fb6b8c7d5f1fac93396c": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "59cfa0a0a402a69a4f5ff3cc8bc735eb50e20cf7edd5926c73a28e60a6882ebf": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b10d27b4cf92b5d792133a7d4b8a02d1a9f80bcdc8249fb375026369a1fd7142": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6494c197d9923de231c42052d8ce6e86f2595dbf58dcb318f3c243e63b40c46a": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d4bfad5460b4f86e2eace88c141c67613b40aab8e8ed3e9fe462461043755454": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8dd30cba050045bc1581d4eef98f867054b89738aad52caa23cad47eb70d4bec": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e87a19a93c5d2dc91d06d11a5c092caee51b44be797a937d1acce1aa7f11dd65": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "11249ba03a564ecf82c055917a951e9dcc4f3dc11ce1a468d1753e29931f2972": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
0s          Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-55b67d5544-kncnd    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "07eba007cfd52e203ca1cf41e5b1557662e2dadfed2be2bbd2f3b6b46e6df17c": add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@caseydavenport @lwr20 @tscswcn @wybaron @tirelibirefe