my pods can't resolve hostname / reach DNS server

[j-beaver]

Hello dear microk8s team
I'm facing an issue with accessing DNS server within pod

==========
see running pods

[alex@snapqa6 ~] k get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/nginx-6799fc88d8-g6ndp 1/1 Running 2 92m
kube-system pod/coredns-7f9c69c78c-rrjlh 1/1 Running 1 79m
kube-system pod/hostpath-provisioner-5c65fbdb4f-m4ksd 1/1 Running 2 10h

==========
go into nginx pod

[alex@snapqa6 ~] k exec -ti pod/nginx-6799fc88d8-g6ndp -- /bin/bash

==========
curl google.com

root@nginx-6799fc88d8-g6ndp:/# curl http://google.com
curl: (6) Could not resolve host: google.com
root@nginx-6799fc88d8-g6ndp:/# exit

==========
iptables are off:

[root@snapqa6 ~]# service iptables status
Redirecting to /bin/systemctl status iptables.service
Unit iptables.service could not be found.

==========
DNS is configured:

[alex@snapqa6 ~]$ microk8s kubectl -n kube-system edit configmap/coredns
Please edit the object below. Lines beginning with a '#' will be ignored,
and an empty file will abort the edit. If an error occurs while saving this file will be
reopened with the relevant failures.

apiVersion: v1
data:
Corefile: ".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n
\ log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa
ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n
\ prometheus :9153\n forward . 16.110.135.51 16.110.135.52 8.8.8.8 \n cache
30\n loop\n reload\n loadbalance\n}\n"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . 16.110.135.51 16.110.135.52 8.8.8.8 \n cache 30\n loop\n reload\n loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
creationTimestamp: "2021-04-26T20:17:07Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
k8s-app: kube-dns
name: coredns
namespace: kube-system
resourceVersion: "1368532"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 6f40bcdc-aefc-42e2-97d4-a3a08db96c2e

[balchua]

coredns is unable to reach the external dns.
I can see from the coredns pod these logs

[ERROR] plugin/errors: 2 4619018533144990871.5358848635072269066. HINFO: read udp 10.1.7.11:60850->16.110.135.51:53: i/o timeout
...
[ERROR] plugin/errors: 2 6765730914986090802.1315597984439123539. HINFO: read udp 10.1.7.11:57479->16.110.135.52:53: i/o timeout
...
[ERROR] plugin/errors: 2 6765730914986090802.1315597984439123539. HINFO: read udp 10.1.7.11:37741->8.8.8.8:53: i/o timeout
...

Maybe you need to open some firewall between this host and your dns servers?

[j-beaver]

hi balchua
thanks for the quick reply

I can reach any of these DNS servers from the host machine:

[alex@snapqa6 ~]$ telnet 16.110.135.51 53
Trying 16.110.135.51...
Connected to 16.110.135.51.
Escape character is '^]'.

the routes are below:
[alex@snapqa6 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 16.17.24.1 0.0.0.0 UG 100 0 0 ens192
10.1.7.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
16.17.24.0 0.0.0.0 255.255.248.0 U 100 0 0 ens192
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

no firewall is running:

[alex@snapqa6 ~]$ service firewalld status
Redirecting to /bin/systemctl status firewalld.service
â—� firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)

[alex@snapqa6 ~]$ service nftables status
Redirecting to /bin/systemctl status nftables.service
â—� nftables.service - Netfilter Tables
Loaded: loaded (/usr/lib/systemd/system/nftables.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:nft(8)

[balchua]

I think i know whats happening. The flannel cni isnt starting. Due to this error in awk

Apr 26 23:40:15 snapqa6.gre.hpecorp.net microk8s.daemon-flanneld[232928]: awk: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

Maybe missing some libraries? Googling a bit, this seems to be related to ncurses. But im not sure. If anyone is familiar with this libtinfo that will be great.

[j-beaver]

thanks balchua
DNS access issue was resolved after ncurses-compat-libs installation