HINFO: unreachable backend: read udp 10.200.0.9:46159->183.60.83.19:53: i/o timeout

[spursy]

I use kubernetes v12, my system is ubuntu 16.

I use the followed command to create DNS resource.

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
bash deploy.sh -i 10.32.0.10 -r "10.32.0.0/24" -s -t coredns.yaml.sed | kubectl apply -f -

After created coredns resource: I check the resources status.

1. check coredns service

kubectl get svc -n kube-system

2. check code DNS pod endpoints

kubectl get ep -n kube-system

3. My DNS config:

cat /etc/recolv.conf

4. Check CoreDNS pod logs

I found CoreDNS pod ip cannot connected to node DNS server ip address.

[chrisohaver]

Can any pod connect to any ip outside of the cluster?
IOW, can you create a pod, and then from that pod, can you connect to an IP that is not hosted in your cluster?

[spursy]

@chrisohaver
I have created a busybox pod, then exec nslookup baidu.com in busybox pod

[vitorreis]

+1 I am having the same issue I believe.

Ubuntu 18.10, just launched a k8s yesterday with calico.
No pod can resolve dns of applications outside the cluster

kubectl  --namespace=kube-system logs coredns-86c58d9df4-sv644
.:53
2019-03-14T23:12:18.074Z [INFO] CoreDNS-1.2.6
2019-03-14T23:12:18.074Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:50264->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:57510->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:32789->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:51126->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:48692->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:60122->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:39497->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:36366->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:39445->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 google.com. AAAA: unreachable backend: read udp 10.42.0.25:52405->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 2023497039190041536.8608779483921538962. HINFO: unreachable backend: read udp 10.42.0.25:42108->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 google.com. A: unreachable backend: read udp 10.42.0.25:44186->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 google.com. A: unreachable backend: read udp 10.42.0.25:57730->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 google.de. AAAA: unreachable backend: read udp 10.42.0.25:60735->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 google.de. A: unreachable backend: read udp 10.42.0.25:41527->213.186.33.99:53: i/o timeout

[chrisohaver]

Can any pod connect to any ip outside of the cluster?
IOW, can you create a pod, and then from that pod, can you connect to an IP that is not hosted in >>your cluster?

I have created a busybox pod, then exec nslookup baidu.com in busybox pod

What I meant is: Please confirm that pods in your cluster can connect to the outside world.
IOW...

1. create a pod
2. From that pod, try to connect to an IP that is not in your cluster. For example, you could try to make a dns query directly to one of your upstream name servers e.g.

nslookup baidu.com. 183.60.83.19

[spursy]

@chrisohaver I have nslookup outside ip from busybox pod, cannot connect.

[chrisohaver]

OK that means that something is wrong with your pod networking in your cluster. Not something specifically wrong with CoreDNS.

Do you have a firewall blocking dns traffic to/from pods?

[vitorreis]

For me I get

kubectl run -it --rm --restart=Never busybox1 --image=busybox sh
If you don't see a command prompt, try pressing enter.
/ # nslookup baidu.com. 183.60.83.19
;; connection timed out; no servers could be reached

/ #

[chrisohaver]

@vitorreis, 183.60.83.19 is @spursy's uptream DNS server. You would want to test on your upstream DNS server...

nslookup google.com. 213.186.33.99

[vitorreis]

@chrisohaver I get the same error for nslookup for google.

Let me share my config (but it's a new cluster bootstrapped this week following of k8s docs).

Here is my coredns deployment:

kube-forro -n kube-system describe deployment coredns
Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Thu, 14 Mar 2019 13:07:16 +0100
Labels:                 k8s-app=kube-dns
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       k8s.gcr.io/coredns:1.2.6
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-86c58d9df4 (2/2 replicas created)
Events:          <none>

and core dns config map

kube-forro -n kube-system describe cm coredns
Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
Corefile:
----
.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       upstream
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    proxy . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Events:  <none>

Services

kube-forro -n kube-system get svc
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
calico-typha              ClusterIP   10.103.141.204   <none>        5473/TCP        26h
kube-dns                  ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   27h
tiller-deploy             ClusterIP   10.108.83.24     <none>        44134/TCP       26h
traefik-ingress-service   NodePort    10.107.226.153   <none>        80:30987/TCP    24h

Endpoints

kube-forro -n kube-system get ep
NAME                      ENDPOINTS                                               AGE
calico-typha              <none>                                                  26h
kube-controller-manager   <none>                                                  27h
kube-dns                  10.42.0.55:53,10.42.0.56:53,10.42.0.55:53 + 1 more...   27h
kube-scheduler            <none>                                                  27h
tiller-deploy             10.42.0.4:44134                                         26h
traefik-ingress-service   91.121.210.212:80                                       24h

/etc/resolv.conf on node machine

root@ubuntu:/home/ubuntu# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 213.186.33.99

CoreDns logs

for p in $(kube-forro get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kube-forro logs --namespace=kube-system $p; done
.:53
2019-03-15T15:16:38.492Z [INFO] CoreDNS-1.2.6
2019-03-15T15:16:38.492Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:33887->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:48492->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:53498->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:57188->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:49250->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:47756->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:40399->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:35771->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:55858->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 7994225720195620561.3842098355147683469. HINFO: unreachable backend: read udp 10.42.0.60:50891->213.186.33.99:53: i/o timeout
.:53
2019-03-15T15:16:38.035Z [INFO] CoreDNS-1.2.6
2019-03-15T15:16:38.035Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:52209->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:59902->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:36795->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:43056->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:47834->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:48029->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 kubernetes.default. A: unreachable backend: read udp 10.42.0.59:35883->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 kubernetes.default. AAAA: unreachable backend: read udp 10.42.0.59:60498->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 kubernetes.default. AAAA: unreachable backend: read udp 10.42.0.59:40426->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 kubernetes.default. A: unreachable backend: read udp 10.42.0.59:44056->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:44335->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:33212->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:60918->213.186.33.99:53: i/o timeout
 [ERROR] plugin/errors: 2 3756344487166424793.1891701709960086559. HINFO: unreachable backend: read udp 10.42.0.59:41406->213.186.33.99:53: i/o timeout

[vitorreis]

update from debugging dns k8s docs section

kube-forro run -it --rm --restart=Never busybox1 --image=busybox sh
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # nslookup kubernetes.default
;; connection timed out; no servers could be reached

/ #

[chrisohaver]

@chrisohaver I get the same error for nslookup for google.

The google part isnt the important change. The important part is that you query your upstream server (213.186.33.99), not @spursy's upstream server.

I suspect that will also timeout/fail. If thats the case, it means your pods also cannot connect to the outside world. So, no fault of CoreDNS, seems like something is globally wrong with pod networking in your cluster. Do you have a firewall blocking dns to/from pods?

[chrisohaver]

@vitorreis, what version of busybox are you using?

[vitorreis]

@chrisohaver

The google part isnt the important change. The important part is that you query your upstream server (213.186.33.99), not @spursy's upstream server.

Thanks for the clarification.

I suspect that will also timeout/fail. If thats the case, it means your pods also cannot connect to the outside world

That's true, no pods can resolve something like ping google.com, however if I run this in the node terminal I can ping successfully, only from the pods it doesn't work.

Do you have a firewall blocking dns to/from pods?

I am running on a fresh machine Ubutu 18.10, no firewall is enabled AFAIK.

root@ubuntu:/home/ubuntu# ufw status
Status: inactive

Is there any other command that I can use to have a check if something is blocking dns from/to pods? Unfortunately I am not a linux expert.

@vitorreis, what version of busybox are you using?

Image: busybox:1.28

[vitorreis]

I did the setup of pod networking using the latest docs available in k8s page.
https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml