New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dns server is null in some pods. #30215

Closed
mdshuai opened this Issue Aug 8, 2016 · 14 comments

Comments

Projects
None yet
10 participants
@mdshuai
Contributor

mdshuai commented Aug 8, 2016

I have two pods in the kube-system namespace, one can resolve dns, When resolve the dns in another pod, it can't resolve the name, and shows the server is null. But the correct dns server already in /etc/resolv.conf

[root@ip-172-18-12-44 heapster]# kubectl get pod --namespace=kube-system
NAME                     READY     STATUS    RESTARTS   AGE
busybox                  1/1       Running   0          10s
heapster-dje5c           1/1       Running   0          1h
influxdb-grafana-iuebs   2/2       Running   0          1h
kube-dns-v19-82nif       3/3       Running   0          2h
[root@ip-172-18-12-44 heapster]# kubectl exec --namespace=kube-system busybox -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
[root@ip-172-18-12-44 heapster]# kubectl exec --namespace=kube-system heapster-dje5c -- nslookup kubernetes.default
Server:    (null)
nslookup: can't resolve 'kubernetes.default': Name does not resolve
Address 1: ::1 localhost
Address 2: 127.0.0.1 localhost

error: error executing remote command: error executing command in container: Error executing in Docker Container: 1
[root@ip-172-18-12-44 heapster]# kubectl exec -it --namespace=kube-system heapster-dje5c -- /bin/sh
/ # cat /etc/resolv.conf 
search kube-system.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 10.0.0.10
options ndots:5

@jay-bhambhani

This comment has been minimized.

Show comment
Hide comment
@jay-bhambhani

jay-bhambhani Aug 9, 2016

I'v run into a very similar error

jay-bhambhani commented Aug 9, 2016

I'v run into a very similar error

@MrHohn

This comment has been minimized.

Show comment
Hide comment
@MrHohn

MrHohn Sep 9, 2016

Member

Heapster is using the 'alpine' base image. It seems like this is a long existing issue for DNS in alpine(gliderlabs/docker-alpine#8).

cc @girishkalele

Member

MrHohn commented Sep 9, 2016

Heapster is using the 'alpine' base image. It seems like this is a long existing issue for DNS in alpine(gliderlabs/docker-alpine#8).

cc @girishkalele

@bprashanth

This comment has been minimized.

Show comment
Hide comment
@bprashanth

bprashanth Sep 9, 2016

Member

Alpine broke dns search paths but fixed it in some later release. It should be fixed in edge, in spite of what https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md indicates. We should update the image to the exact version with the fix, which you can figure out with a test like (or prove im wrong by running the test on edge):
docker run --dns 8.8.8.8 --dns-search github.com -it alpine:3.3 nslookup www

Member

bprashanth commented Sep 9, 2016

Alpine broke dns search paths but fixed it in some later release. It should be fixed in edge, in spite of what https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md indicates. We should update the image to the exact version with the fix, which you can figure out with a test like (or prove im wrong by running the test on edge):
docker run --dns 8.8.8.8 --dns-search github.com -it alpine:3.3 nslookup www

@bprashanth

This comment has been minimized.

Show comment
Hide comment
@bprashanth

bprashanth Sep 9, 2016

Member

@piosz @mwielgus for the alpine issue

Member

bprashanth commented Sep 9, 2016

@piosz @mwielgus for the alpine issue

@jay-bhambhani

This comment has been minimized.

Show comment
Hide comment
@jay-bhambhani

jay-bhambhani Sep 9, 2016

This is happening for us running the Ubuntu base image

jay-bhambhani commented Sep 9, 2016

This is happening for us running the Ubuntu base image

@bprashanth

This comment has been minimized.

Show comment
Hide comment
@bprashanth

bprashanth Sep 9, 2016

Member

what's you resolv.conf look like? ubuntu doesn't suffer from the same issue so it's likely something wrong with services or connectivity, see http://kubernetes.io/docs/user-guide/debugging-services/#does-the-service-work-by-dns for debugging tips

Member

bprashanth commented Sep 9, 2016

what's you resolv.conf look like? ubuntu doesn't suffer from the same issue so it's likely something wrong with services or connectivity, see http://kubernetes.io/docs/user-guide/debugging-services/#does-the-service-work-by-dns for debugging tips

@piosz

This comment has been minimized.

Show comment
Hide comment
@piosz

piosz Sep 12, 2016

Member

I created an issue in Heapster to migrate out of Apline kubernetes/heapster#1284

Member

piosz commented Sep 12, 2016

I created an issue in Heapster to migrate out of Apline kubernetes/heapster#1284

@cordoval

This comment has been minimized.

Show comment
Hide comment
@cordoval

cordoval Jun 6, 2017

i ran into this:

~ kubectl exec -it nginx-deployment-4178600648-mfbpx bash                                                                                                                                                                               iMac [9:36:12]
bash-4.3# nslookup kubernetes.default
nslookup: can’t resolve ‘(null)‘: Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

also

ps auxw|grep kubelet

returns nothing

~ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns                                                                                                                                                                         iMac [10:01:32]
NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-1321724180-37hml   3/3       Running   0          15h
kube-dns-1321724180-4065v   0/3       Evicted   0          15h
kube-dns-1321724180-4q34m   0/3       Evicted   0          15h
kube-dns-1321724180-59w8x   0/3       Evicted   0          15h
kube-dns-1321724180-5qflv   0/3       Evicted   0          15h
kube-dns-1321724180-5rfjj   0/3       Evicted   0          15h
kube-dns-1321724180-6dwx8   0/3       Evicted   0          15h
kube-dns-1321724180-73k1g   0/3       Evicted   0          15h
kube-dns-1321724180-811kn   0/3       Evicted   0          15h
kube-dns-1321724180-9ndph   0/3       Evicted   0          7d
kube-dns-1321724180-b1b9v   0/3       Evicted   0          15h
kube-dns-1321724180-d7ls6   0/3       Evicted   0          7d
kube-dns-1321724180-hsll1   0/3       Evicted   0          15h
kube-dns-1321724180-kd7wt   0/3       Evicted   0          15h
kube-dns-1321724180-ldjjc   0/3       Evicted   0          15h
kube-dns-1321724180-m6nmf   0/3       Evicted   0          15h
kube-dns-1321724180-mqx9z   0/3       Evicted   0          15h
kube-dns-1321724180-ndsx3   0/3       Evicted   0          15h
kube-dns-1321724180-nfrzd   3/3       Running   0          23h
kube-dns-1321724180-pk2gj   0/3       Evicted   0          15h
kube-dns-1321724180-qxqvc   0/3       Evicted   0          15h
kube-dns-1321724180-r8jq0   0/3       Evicted   0          15h
kube-dns-1321724180-vvkn0   0/3       Evicted   0          15h
kube-dns-1321724180-w35dg   0/3       Evicted   0          15h
kube-dns-1321724180-znx21   0/3       Evicted   0          15h

see the 3 dns running 3/3 at the same time? nvm

i checked the logs of dns and why it crashed and here:

container "kubedns" in pod "kube-dns-1321724180-5qflv" is not available
or

2017-06-06T15:13:51.49485473Z I0606 15:13:51.494533       1 dns.go:264] New service: my-service
2017-06-06T15:13:51.494886075Z I0606 15:13:51.494622       1 dns.go:462] Added SRV record &{Host:my-service.default.svc.cluster.local. Port:9000 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.49489118Z I0606 15:13:51.494654       1 dns.go:264] New service: nginx-service
2017-06-06T15:13:51.494894928Z I0606 15:13:51.494683       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:80 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.4948985Z I0606 15:13:51.494704       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494902203Z I0606 15:13:51.494732       1 dns.go:264] New service: kubernetes-dashboard
2017-06-06T15:13:51.494906642Z I0606 15:13:51.494759       1 dns.go:264] New service: kubernetes
2017-06-06T15:13:51.494909907Z I0606 15:13:51.494785       1 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494913664Z I0606 15:13:51.494813       1 dns.go:264] New service: kube-dns
2017-06-06T15:13:51.495192292Z I0606 15:13:51.494838       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495201152Z I0606 15:13:51.494859       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495204827Z I0606 15:13:51.494882       1 dns.go:264] New service: redis-service
2017-06-06T15:13:51.495208133Z I0606 15:13:51.494903       1 dns.go:462] Added SRV record &{Host:redis-service.default.svc.cluster.local. Port:6379 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495211637Z I0606 15:13:51.494924       1 dns.go:264] New service: postgres-service
2017-06-06T15:13:51.495215117Z I0606 15:13:51.494943       1 dns.go:462] Added SRV record &{Host:postgres-service.default.svc.cluster.local. Port:5432 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}

cordoval commented Jun 6, 2017

i ran into this:

~ kubectl exec -it nginx-deployment-4178600648-mfbpx bash                                                                                                                                                                               iMac [9:36:12]
bash-4.3# nslookup kubernetes.default
nslookup: can’t resolve ‘(null)‘: Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

also

ps auxw|grep kubelet

returns nothing

~ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns                                                                                                                                                                         iMac [10:01:32]
NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-1321724180-37hml   3/3       Running   0          15h
kube-dns-1321724180-4065v   0/3       Evicted   0          15h
kube-dns-1321724180-4q34m   0/3       Evicted   0          15h
kube-dns-1321724180-59w8x   0/3       Evicted   0          15h
kube-dns-1321724180-5qflv   0/3       Evicted   0          15h
kube-dns-1321724180-5rfjj   0/3       Evicted   0          15h
kube-dns-1321724180-6dwx8   0/3       Evicted   0          15h
kube-dns-1321724180-73k1g   0/3       Evicted   0          15h
kube-dns-1321724180-811kn   0/3       Evicted   0          15h
kube-dns-1321724180-9ndph   0/3       Evicted   0          7d
kube-dns-1321724180-b1b9v   0/3       Evicted   0          15h
kube-dns-1321724180-d7ls6   0/3       Evicted   0          7d
kube-dns-1321724180-hsll1   0/3       Evicted   0          15h
kube-dns-1321724180-kd7wt   0/3       Evicted   0          15h
kube-dns-1321724180-ldjjc   0/3       Evicted   0          15h
kube-dns-1321724180-m6nmf   0/3       Evicted   0          15h
kube-dns-1321724180-mqx9z   0/3       Evicted   0          15h
kube-dns-1321724180-ndsx3   0/3       Evicted   0          15h
kube-dns-1321724180-nfrzd   3/3       Running   0          23h
kube-dns-1321724180-pk2gj   0/3       Evicted   0          15h
kube-dns-1321724180-qxqvc   0/3       Evicted   0          15h
kube-dns-1321724180-r8jq0   0/3       Evicted   0          15h
kube-dns-1321724180-vvkn0   0/3       Evicted   0          15h
kube-dns-1321724180-w35dg   0/3       Evicted   0          15h
kube-dns-1321724180-znx21   0/3       Evicted   0          15h

see the 3 dns running 3/3 at the same time? nvm

i checked the logs of dns and why it crashed and here:

container "kubedns" in pod "kube-dns-1321724180-5qflv" is not available
or

2017-06-06T15:13:51.49485473Z I0606 15:13:51.494533       1 dns.go:264] New service: my-service
2017-06-06T15:13:51.494886075Z I0606 15:13:51.494622       1 dns.go:462] Added SRV record &{Host:my-service.default.svc.cluster.local. Port:9000 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.49489118Z I0606 15:13:51.494654       1 dns.go:264] New service: nginx-service
2017-06-06T15:13:51.494894928Z I0606 15:13:51.494683       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:80 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.4948985Z I0606 15:13:51.494704       1 dns.go:462] Added SRV record &{Host:nginx-service.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494902203Z I0606 15:13:51.494732       1 dns.go:264] New service: kubernetes-dashboard
2017-06-06T15:13:51.494906642Z I0606 15:13:51.494759       1 dns.go:264] New service: kubernetes
2017-06-06T15:13:51.494909907Z I0606 15:13:51.494785       1 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.494913664Z I0606 15:13:51.494813       1 dns.go:264] New service: kube-dns
2017-06-06T15:13:51.495192292Z I0606 15:13:51.494838       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495201152Z I0606 15:13:51.494859       1 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495204827Z I0606 15:13:51.494882       1 dns.go:264] New service: redis-service
2017-06-06T15:13:51.495208133Z I0606 15:13:51.494903       1 dns.go:462] Added SRV record &{Host:redis-service.default.svc.cluster.local. Port:6379 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2017-06-06T15:13:51.495211637Z I0606 15:13:51.494924       1 dns.go:264] New service: postgres-service
2017-06-06T15:13:51.495215117Z I0606 15:13:51.494943       1 dns.go:462] Added SRV record &{Host:postgres-service.default.svc.cluster.local. Port:5432 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano

cmluciano Jun 6, 2017

Member

@cordoval It looks like your node is under significant pressure. All of your entries seem to be evicted. Is the node currently under memory pressure?

Member

cmluciano commented Jun 6, 2017

@cordoval It looks like your node is under significant pressure. All of your entries seem to be evicted. Is the node currently under memory pressure?

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano

cmluciano Jun 6, 2017

Member

/sig network
/area dns
/assign

Member

cmluciano commented Jun 6, 2017

/sig network
/area dns
/assign

@cordoval

This comment has been minimized.

Show comment
Hide comment
@cordoval

cordoval Jun 6, 2017

what do you mean on your last comments? also i killed a node and the ASG resized them and it now works, maybe kube-ip-proxy got down for some reason and that was the cause, now i see everything working fine.

cordoval commented Jun 6, 2017

what do you mean on your last comments? also i killed a node and the ASG resized them and it now works, maybe kube-ip-proxy got down for some reason and that was the cause, now i see everything working fine.

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano

cmluciano Jun 6, 2017

Member

@piosz @bprashanth Do you think we can close this issue now?

Member

cmluciano commented Jun 6, 2017

@piosz @bprashanth Do you think we can close this issue now?

@MrHohn

This comment has been minimized.

Show comment
Hide comment
@MrHohn

MrHohn Jun 6, 2017

Member

@cmluciano The original post is about the nslookup bug in Alpine image and that is cleared. I think we should close this one to avoid stacking up other unrelated issues in the same place.

Member

MrHohn commented Jun 6, 2017

@cmluciano The original post is about the nslookup bug in Alpine image and that is cleared. I think we should close this one to avoid stacking up other unrelated issues in the same place.

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano

cmluciano Jun 6, 2017

Member

/close

Member

cmluciano commented Jun 6, 2017

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment