New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-dns copies 127.0.0.35 from host's /etc/resolv.conf, doesn't work #45828

Closed
gjcarneiro opened this Issue May 15, 2017 · 14 comments

Comments

Projects
None yet
7 participants
@gjcarneiro

gjcarneiro commented May 15, 2017

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kube-dns resolv.conf


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): v1.6.3

Environment: ubuntu 17.04

  • Cloud provider or hardware configuration: baremetal i7 pc, 16gb

  • OS (e.g. from /etc/os-release): ubuntu 17.04

  • Kernel (e.g. uname -a):4.10.0-21-generic

  • Install tools: kubeadm

  • Others:

What happened:

After installing on baremetal ubuntu 17.04 (my desktop), kube-dns not working

What you expected to happen:

dns should work

How to reproduce it (as minimally and precisely as possible):

Follow the kubeadm install steps on ubuntu 17.04

Anything else we need to know:

As far as I was able to track down (and temporarily solve), my ubuntu desktop has an /etc/resolv.conf file that is autogenerated to look like this:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

When kube-dns starts, the dnsmasq container somehow ignores the --dns options given to the docker daemon, and instead copies my desktop's /etc/resolv.conf file.

Obviously, 127.0.0.35 doesn't work inside the pods. It should never have been used in the first place. There's a reason why docker daemon has --dns command line options. These options should be used. Docker does the right thing, don't override it.

@gjcarneiro gjcarneiro changed the title from kube-dns copies 127.0.0.35 from host's /etc/resolve.conf, doesn't work to kube-dns copies 127.0.0.35 from host's /etc/resolv.conf, doesn't work May 15, 2017

@cmluciano

This comment has been minimized.

Member

cmluciano commented May 16, 2017

/area dns

@cmluciano

This comment has been minimized.

Member

cmluciano commented May 16, 2017

/sig network

@jmzwcn

This comment has been minimized.

jmzwcn commented May 18, 2017

Could I please take a try? any suggestion?

@antoineco

This comment has been minimized.

Contributor

antoineco commented May 18, 2017

I think you're looking for the kubelet --cluster-dns flag. (kubelet doc)
The --dns flag on the Docker deamon is ignored since #29378 was merged.

@thockin

This comment has been minimized.

Member

thockin commented May 19, 2017

You can pass --resolv-conf=<path> to kubelet to read a different resolv.conf file with values you want.

@thockin thockin closed this May 19, 2017

@gjcarneiro

This comment has been minimized.

gjcarneiro commented May 19, 2017

Maybe kubeadm needs to be smarter?... It bothers me that the default install doesn't work. This should be fixed somewhere IMHO...

@antoineco

This comment has been minimized.

Contributor

antoineco commented May 19, 2017

The fact that you're using systemd-resolved is a particular case (this would also be true if you'd be using dnsmasq), most machines are configured to send DNS lookups to a host which is not localhost, so mounting the resolv.conf totally makes sense.

Even if kubeadm would detect that you're using a local resolver, how would it cope with that? How is it supposed to know which nameserver to use instead? Should it default to 8.8.8.8 and break the resolution of local names? I don't think so.

Note: even when using kubeadm you can edit the generated systemd drop-in for kubelet.service manually at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. See https://kubernetes.io/docs/getting-started-guides/kubeadm/

@gjcarneiro

This comment has been minimized.

gjcarneiro commented May 19, 2017

The fact that you're using systemd-resolved is a particular case

Well, my "particular case" is a standard Ubuntu desktop install. Yes, I grant you that no person in their right mind would run Network Manager in a Ubuntu Server, but this is Desktop, it is normal.

If you think it's OK for kubeadm to not install correctly by default to a Ubuntu desktop, only server, I accept it, but it's a pity, Kubernetes should try harder be easy to install in a developer box. And, no, minikube is not a good solution, due to numerous vboxsf filesystem bugs which make host files volume mounting frustrating, also due to the added memory consumption and cpu overhead of virtualbox.

BTW, the workaround I've used was to create a kube-dns ConfigMap with a upstreamNameservers entry.

@antoineco

This comment has been minimized.

Contributor

antoineco commented May 19, 2017

It's not that uncommon I agree, any distribution using systemd can potentially default to using the systemd-resolved stub DNS resolver (btw this has nothing to do with NetworkManager). I meant that it is delicate to handle local resolvers.

Can you check the content of this file on your machine? /run/systemd/resolve/resolv.conf
Does it contain any entry?

@gjcarneiro

This comment has been minimized.

gjcarneiro commented May 19, 2017

My /run/systemd/resolve/resolv.conf contains my actual DNS configurations, the ones I entered via the Network Manager UI (/usr/bin/nm-connection-editor):

# This file is managed by systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known DNS servers.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.122.65.250
nameserver 10.122.65.251
nameserver 10.122.65.240
search gambit int.gambit

The above configuration, if it were to be copied into kube-dns, should work.

@antoineco

This comment has been minimized.

Contributor

antoineco commented May 19, 2017

So, as mentioned by @thockin, setting --resolv-conf=/run/systemd/resolve/resolv.conf should be a decent workaround for now.

How about raising a feature request in the kubeadm repo to default to that file in case systemd-resolved is running with DNSStubListener and /etc/resolv.conf is configured with the local resolver?

@gjcarneiro

This comment has been minimized.

gjcarneiro commented May 19, 2017

That seems reasonable. Created kubernetes/kubeadm#273. Thanks.

@andrewrothman

This comment has been minimized.

andrewrothman commented Dec 3, 2017

I just ran into this same issue. As @thockin and @antoineco mentioned, I edited /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.

I changed this line:

Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

to this:

Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --resolv-conf=/run/systemd/resolve/resolv.conf

Which worked perfectly, and my pods can now access the internet as normal:

ping google.com # works now

Maybe adding a check to kubeadm init that reads the host's /etc/resolv.conf and adds this additional arg if the nameserver is set to 127.0.0.53 would solve the problem?

@andrewrothman

This comment has been minimized.

andrewrothman commented Jan 9, 2018

I'm sure there are other factors at play that I'm not aware of, but I'd just like to report that my pods are able to communicate out to the internet as well as to other pods in the cluster via DNS hostname (I haven't tried exposed services yet), and I haven't had networking issues since changing my config to the value I mentioned above.

Again, I'm sure there are complexities I'm not aware of. Just wanted to report my own experience and I'd be happy to answer any debugging questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment