kubedns dnsmasq pod fails with blank IP for the DNS host

[sgmiller]

Can't get a cluster started using kubeadm on v1.6.0-beta1 (kube-dns v1.14.1). The dnsmasq pod fails with:

I0308 16:30:25.353537       1 nanny.go:108] dnsmasq[11]: bad address at /etc/hosts line 7
I0308 16:30:25.353540       1 nanny.go:108] dnsmasq[11]: read /etc/hosts - 6 addresses

/etc/hosts contains:

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
	kube-dns-1630391940-grt67

So presumably it can't get the information it needs from the ConfigMap?

[MrHohn]

kube-dns ConfgiMap won't affect /etc/hosts file. Instead, kubelet is responsible to setting the hosts file up (https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/kubelet_pods.go#L186-L207).

And seems like you've got a empty pod IP for this kube-dns pod. Can you also check the output of $ kubectl get pods KUBE_DNS_POD_NAME --namespace=kube-system -o yaml? Are the other pods having the same issue?

It would be helpful if you could also check kubelet's logs on the node where kube-dns pod is on.

[sgmiller]

I see. Other pods don't seem to have this problem. Looking on kube-proxy for example has a complete hosts file.

The kube-dns pod's yaml is attached.
kube-dns.txt

[sgmiller]

These lines from syslog on the master (the only node so far) are interesting:

 Write status for kube-dns-1630391940-bnl7d/kube-system: &container.PodStatus{ID:"43e3338c-042a-11e7-a063-02ead988c3bd", Name:"kube-dns-1630391940-bnl7d", Namespace:"kube-system", IP:"", ContainerStatuses:[]*container.ContainerStatus{(*container.ContainerStatus)(0xc421696000), (*container.ContainerStatus)(0xc42118c7e0), (*container.ContainerStatus)(0xc421696380), (*container.ContainerStatus)(0xc421696620), (*container.ContainerStatus)(0xc42118c9a0)}, SandboxStatuses:[]*runtime.PodSandboxStatus{(*runtime.PodSandboxStatus)(0xc42137fb80)}} (err: <nil>)

 NetworkPlugin cni failed on the status hook for pod "kube-dns-1630391940-bnl7d_kube-system": Unexpected command output Device "eth0" does not exist.

[sgmiller]

And indeed, ifconfig in the kubedns container has only "lo":

# kubectl exec kube-dns-1630391940-bnl7d -c kubedns ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:732 errors:0 dropped:0 overruns:0 frame:0
          TX packets:732 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:64918 (63.3 KiB)  TX bytes:64918 (63.3 KiB)

[MrHohn]

Hmm, could you check other pods that are not using hostNetwork (kube-proxy set hostNetwork=true)? From your yaml file it seems like it does not have podIP allocated. I wonder is the underlying network broken?

Here is a similar issue: https://github.com/projectcalico/calicoctl/issues/1407

[sgmiller]

Close. Due to a docker bug, the dns pod couldn't be spun up (couldn't write memory limits properly). Fixing that, I get a little closer, but it can't seem to reach what I'm guessing is the API server. kubeadm sets this up on port 6443, but this isn't what kube-dns is trying to reach:

I0308 19:02:15.998889       1 dns.go:174] DNS server not ready, retry in 500 milliseconds
E0308 19:02:16.017504       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0308 19:02:16.017504       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable

[MrHohn]

Yeah, it was trying to reach the apiserver through the default kubernetes service but failed.

Not sure how kubeadm set this up, but as I know kube-dns either use the kubecfg file specified by --kubecfg-file and kube-master-url flags or use the env vars (KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, which should be set up automatically when pod is created) as the configurations when creating its k8s client.

[sgmiller]

Its the env vars, and they're wrong. What controls setting those?

[MrHohn]

These values are piped into kubelet through --kubeconfig or --api-servers flags.

I'm not pretty sure if the real issue is about "incorrect kubernetes service IP/port" or not. Feel like there may be some mechanism in kubeadm I misunderstood. You may want to open an issue against kubeadm to seek for the real root cause.

[MrHohn]

Oops, looks like I am wrong. Those env vars are generated from services themselves but not from flags:
https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/kubelet_pods.go#L389
https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/envvars/envvars.go#L27-L60

[sgmiller]

I agree, I don't think kube-dns has anything to do with this, closing.