Dns server is null in some pods.

[mdshuai]

I have two pods in the kube-system namespace, one can resolve dns, When resolve the dns in another pod, it can't resolve the name, and shows the server is null. But the correct dns server already in /etc/resolv.conf

[root@ip-172-18-12-44 heapster]# kubectl get pod --namespace=kube-system
NAME                     READY     STATUS    RESTARTS   AGE
busybox                  1/1       Running   0          10s
heapster-dje5c           1/1       Running   0          1h
influxdb-grafana-iuebs   2/2       Running   0          1h
kube-dns-v19-82nif       3/3       Running   0          2h
[root@ip-172-18-12-44 heapster]# kubectl exec --namespace=kube-system busybox -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
[root@ip-172-18-12-44 heapster]# kubectl exec --namespace=kube-system heapster-dje5c -- nslookup kubernetes.default
Server:    (null)
nslookup: can't resolve 'kubernetes.default': Name does not resolve
Address 1: ::1 localhost
Address 2: 127.0.0.1 localhost

error: error executing remote command: error executing command in container: Error executing in Docker Container: 1
[root@ip-172-18-12-44 heapster]# kubectl exec -it --namespace=kube-system heapster-dje5c -- /bin/sh
/ # cat /etc/resolv.conf 
search kube-system.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 10.0.0.10
options ndots:5

[jay-bhambhani]

I'v run into a very similar error

[MrHohn]

Heapster is using the 'alpine' base image. It seems like this is a long existing issue for DNS in alpine(gliderlabs/docker-alpine#8).

cc @girishkalele

[bprashanth]

Alpine broke dns search paths but fixed it in some later release. It should be fixed in edge, in spite of what https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md indicates. We should update the image to the exact version with the fix, which you can figure out with a test like (or prove im wrong by running the test on edge):
docker run --dns 8.8.8.8 --dns-search github.com -it alpine:3.3 nslookup www

[bprashanth]

@piosz @mwielgus for the alpine issue

[jay-bhambhani]

This is happening for us running the Ubuntu base image

[bprashanth]

what's you resolv.conf look like? ubuntu doesn't suffer from the same issue so it's likely something wrong with services or connectivity, see http://kubernetes.io/docs/user-guide/debugging-services/#does-the-service-work-by-dns for debugging tips

[piosz]

I created an issue in Heapster to migrate out of Apline kubernetes-retired/heapster#1284

[cordoval]

i ran into this:

~ kubectl exec -it nginx-deployment-4178600648-mfbpx bash                                                                                                                                                                               iMac [9:36:12]
bash-4.3# nslookup kubernetes.default
nslookup: can’t resolve ‘(null)‘: Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

also

ps auxw|grep kubelet

returns nothing

~ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns                                                                                                                                                                         iMac [10:01:32]
NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-1321724180-37hml   3/3       Running   0          15h
kube-dns-1321724180-4065v   0/3       Evicted   0          15h
kube-dns-1321724180-4q34m   0/3       Evicted   0          15h
kube-dns-1321724180-59w8x   0/3       Evicted   0          15h
kube-dns-1321724180-5qflv   0/3       Evicted   0          15h
kube-dns-1321724180-5rfjj   0/3       Evicted   0          15h
kube-dns-1321724180-6dwx8   0/3       Evicted   0          15h
kube-dns-1321724180-73k1g   0/3       Evicted   0          15h
kube-dns-1321724180-811kn   0/3       Evicted   0          15h
kube-dns-1321724180-9ndph   0/3       Evicted   0          7d
kube-dns-1321724180-b1b9v   0/3       Evicted   0          15h
kube-dns-1321724180-d7ls6   0/3       Evicted   0          7d
kube-dns-1321724180-hsll1   0/3       Evicted   0          15h
kube-dns-1321724180-kd7wt   0/3       Evicted   0          15h
kube-dns-1321724180-ldjjc   0/3       Evicted   0          15h
kube-dns-1321724180-m6nmf   0/3       Evicted   0          15h
kube-dns-1321724180-mqx9z   0/3       Evicted   0          15h
kube-dns-1321724180-ndsx3   0/3       Evicted   0          15h
kube-dns-1321724180-nfrzd   3/3       Running   0          23h
kube-dns-1321724180-pk2gj   0/3       Evicted   0          15h
kube-dns-1321724180-qxqvc   0/3       Evicted   0          15h
kube-dns-1321724180-r8jq0   0/3       Evicted   0          15h
kube-dns-1321724180-vvkn0   0/3       Evicted   0          15h
kube-dns-1321724180-w35dg   0/3       Evicted   0          15h
kube-dns-1321724180-znx21   0/3       Evicted   0          15h

see the 3 dns running 3/3 at the same time? nvm

i checked the logs of dns and why it crashed and here:

container "kubedns" in pod "kube-dns-1321724180-5qflv" is not available
or

2017-06-06T15:13:51.49485473Z I0606 15:13:51.494533       1 dns.go:264] New service: my-service
2017-06-06T15:13:51.494886075Z I0606 15:13:51.494622       1 dns.go:462] Added SRV record &{Host:my-service.default.svc.cluster.local. Port:9000 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.49489118Z I0606 15:13:51.494654       1 dns.go:264] New service: nginx-service
2017-06-06T15:13:51.494894928Z I0606 15:13:51.494683       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:80 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.4948985Z I0606 15:13:51.494704       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494902203Z I0606 15:13:51.494732       1 dns.go:264] New service: kubernetes-dashboard
2017-06-06T15:13:51.494906642Z I0606 15:13:51.494759       1 dns.go:264] New service: kubernetes
2017-06-06T15:13:51.494909907Z I0606 15:13:51.494785       1 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494913664Z I0606 15:13:51.494813       1 dns.go:264] New service: kube-dns
2017-06-06T15:13:51.495192292Z I0606 15:13:51.494838       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495201152Z I0606 15:13:51.494859       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495204827Z I0606 15:13:51.494882       1 dns.go:264] New service: redis-service
2017-06-06T15:13:51.495208133Z I0606 15:13:51.494903       1 dns.go:462] Added SRV record &{Host:redis-service.default.svc.cluster.local. Port:6379 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495211637Z I0606 15:13:51.494924       1 dns.go:264] New service: postgres-service
2017-06-06T15:13:51.495215117Z I0606 15:13:51.494943       1 dns.go:462] Added SRV record &{Host:postgres-service.default.svc.cluster.local. Port:5432 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}

[cmluciano]

@cordoval It looks like your node is under significant pressure. All of your entries seem to be evicted. Is the node currently under memory pressure?

[cmluciano]

/sig network
/area dns
/assign

[cordoval]

what do you mean on your last comments? also i killed a node and the ASG resized them and it now works, maybe kube-ip-proxy got down for some reason and that was the cause, now i see everything working fine.

[cmluciano]

@piosz @bprashanth Do you think we can close this issue now?

[MrHohn]

@cmluciano The original post is about the nslookup bug in Alpine image and that is cleared. I think we should close this one to avoid stacking up other unrelated issues in the same place.

[cmluciano]

/close