Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Public domain names cannot be resolved if host uses DHCP #31337

Closed
cheld opened this issue Aug 24, 2016 · 10 comments

Comments

@cheld
Copy link
Member

commented Aug 24, 2016

BUG REPORT

Kubernetes version (use kubectl version): v1.3.5

Environment:
Hyperkube with default settings: https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode
Host: Ubuntu 15

What happened:

kubectl run test --image=busybox -- sleep 100000
kubectl exec test-1954845456-iv7ld nslookup www.heise.de

fails, but

kubectl exec test-1954845456-iv7ld nslookup www.heise.de 8.8.8.8

works.

Anything else do we need to know:
I investigated a bit. The following steps describe what happens:

  1. Name look-ups are forwarded to sky-dns
  2. If sky-dns cannot resolve it, the request will be forwarded.
  3. The source for the forwarding nameserver is /etc/resolv.conf in the sky-dns container
  4. The contents of /etc/resov.conf in the container is the same as on host
  5. In my case the nameserver on host is 127.0.1.1 (dnsmasq configured by DHCP)

If I change the nameserver on the host to 8.8.8.8 and start the cluster everything works fine. One other option to solve the problem is to set a nameserver forwarding flag for sky-dns

Is this the intended behavior?

FYI @luxas, @zreigz

@luxas

This comment has been minimized.

Copy link
Member

commented Aug 24, 2016

I also came across this, but I don't know if it works in the same way with "normal" kubelet; that the host's /etc/resolv.conf file must include a publicly available DNS server like 8.8.8.8

cc @kubernetes/sig-network @thockin @ArtfulCoder @madhusudancs

@madhusudancs

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2016

@cheld this is the intended behavior. Also, see the kubelet's --resolv-conf= flag - http://kubernetes.io/docs/admin/kubelet. You can provide your own base resolv.conf to kubelet, which will in turn be used to generate the KubeDNS forwarding nameservers. Otherwise, host's /etc/resolv.conf file will be used.

I don't think the current version of KubeDNS allows --nameserver flag. We could probably add it if it is necessary.

@madhusudancs

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2016

@luxas it's not necessary that the nameserver specified in the resolv.conf is publicly available, it should be reachable by KubeDNS, or in other words it should be reachable by the pods in your cluster.

@cheld

This comment has been minimized.

Copy link
Member Author

commented Aug 25, 2016

@madhusudancs

Ok I see. But it should work by default somehow. Docker seems to replace 127.* silently with 8.8.8.8

$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search xxxxxx.fujitsu.com

$ docker run busybox cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search xxxxx.fujitsu.com

nameserver 8.8.8.8
nameserver 8.8.4.4

We could do the same. Maybe optional. WDYT

@thockin

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

It's a little magical to do that - if your environment has a real DNS
server, this will bypass it. I'd rather log an event on the pod and fail
to run it, or better, translate 127.0.0.0/* into the node's IP or something
like that...

On Thu, Aug 25, 2016 at 4:30 AM, Christoph Held notifications@github.com
wrote:

@madhusudancs https://github.com/madhusudancs

Ok I see. But it should work by default somehow. Docker seems to replace
127.* silently with 8.8.8.8

$ cat /etc/resolv.conf

Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 127.0.1.1
search xxxxxx.fujitsu.com

$ docker run busybox cat /etc/resolv.conf

Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

search xxxxx.fujitsu.com

nameserver 8.8.8.8
nameserver 8.8.4.4

We could do the same in skydns. Maybe optional. WDYT


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#31337 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGvFKsfoigSD3d43MaPQr2nb9QgLks5qjXzwgaJpZM4Jr2I1
.

@cheld

This comment has been minimized.

Copy link
Member Author

commented Aug 26, 2016

@thockin a resolve for a company internal DNS would not work, but neither does it work right now....

Host IP would be very nice, but it is not working on my machine. I assume it requires configuration changes on host to make dnsmasq accept connections.

$ ip -4 a show dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 172.31.0.87/24 brd 172.31.0.255 scope global dynamic enp0s25
       valid_lft 85506sec preferred_lft 85506sec

$ kubectl run test3 --image=busybox sleep 100000
$ kubectl exec test3-3084950299-w9qz6 nslookup www.heise.de 172.31.0.87
nslookup: can't resolve 'www.heise.de'
Server:    172.31.0.87
Address 1: 172.31.0.87

However, kubelet has access to the host and could determine the DNS server returned by DHCP and generate a resolv.conf file to be used.

@thockin

This comment has been minimized.

Copy link
Member

commented Aug 26, 2016

Kubelet can not tap into DHCP - that's way too wide-open. You can have the
DHCP pipeline write a resolv.conf file for kubelet to use...

On Thu, Aug 25, 2016 at 11:42 PM, Christoph Held notifications@github.com
wrote:

@thockin https://github.com/thockin a resolve for a company internal
DNS would not work, but neither does it work right now....

Host IP would be very nice, but it is not working on my machine. I assume
it requires configuration changes on host to make dnsmasq accept
connections.

$ ip -4 a show dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.31.0.87/24 brd 172.31.0.255 scope global dynamic enp0s25
valid_lft 85506sec preferred_lft 85506sec

$ kubectl run test3 --image=busybox sleep 100000
$ kubectl exec test3-3084950299-w9qz6 nslookup www.heise.de 172.31.0.87
nslookup: can't resolve 'www.heise.de'
Server: 172.31.0.87
Address 1: 172.31.0.87

However, kubelet has access to the host and could determine the DNS server
returned by DHCP and generate a resolv.conf file to be used.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#31337 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVCWSOMtv2dYWu7ksMmMw3mLZJ1rbks5qjorMgaJpZM4Jr2I1
.

@cheld

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2016

@thockin I thought about parsing a file or using a tool like nmcli. But I do not have something clever in mind.

I did some additional testing and started a centos and suse VM. Actually, those write a proper IP to /etc/resolve.conf. Maybe it is only a problem of Ubuntu laptop installation.

So, do we have any good idea to proceed with the issue or should I close it?

@thockin

This comment has been minimized.

Copy link
Member

commented Aug 29, 2016

I don't think the solution can come from within Kubernetes - it is too
specific to the exact installation.

On Mon, Aug 29, 2016 at 4:44 AM, Christoph Held notifications@github.com
wrote:

@thockin https://github.com/thockin I thought about parsing a file or
using a tool like nmcli. But I do not have something clever in mind.

I did some additional testing and started a centos and suse VM. Actually,
those write a proper IP to /etc/resolve.conf. Maybe it is only a problem of
Ubuntu laptop installation.

So, do we have any good idea to proceed with the issue or should I close
it?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#31337 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVM2dBxZuawJOMcbYKSnS8UMuVwT6ks5qksY3gaJpZM4Jr2I1
.

@cheld

This comment has been minimized.

Copy link
Member Author

commented Aug 30, 2016

Ok, thanks for comments

@cheld cheld closed this Aug 30, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.