New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu 18.04 + Docker 17.12.1-ce break DNS resolution #2187

Closed
mprobst opened this Issue Jun 18, 2018 · 17 comments

Comments

Projects
None yet
@mprobst

mprobst commented Jun 18, 2018

This might be a dupe of #1654 (which is closed - but then this is still happening). /CC @sanimej who's been looking at that last year.

The default installation of Docker on Ubuntu 18.04 is broken if Google DNS servers 8.8.8.8 and 8.8.4.4 are not reachable for some reason.

Out of the box Ubuntu runs systemd, which creates a nameserver running on localhost:

$ docker --version
Docker version 17.12.1-ce, build 7390fc6
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]

nameserver 127.0.0.53

Docker ignores the localhost nameserver:

$ docker run  -ti busybox cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
# [...]
nameserver 8.8.8.8
nameserver 8.8.4.4

Because on my network, 8.8.8.8 is not reachable for some reason (different bug), name lookups time out:

$ docker run  -ti busybox nslookup www.google.com
Server:    8.8.8.8
Address 1: 8.8.8.8

nslookup: can't resolve 'www.google.com'

For completeness sake, the symptom that lead me to find this is that when building a Go based image and running go get in a build step, I get a Could not resolve host: github.com (took me a while to piece this one together...):

Step 6/9 : RUN go get ./
 ---> Running in f64c6b120d88
# cd .; git clone https://github.com/julienschmidt/httprouter /go/src/github.com/julienschmidt/httprouter
Cloning into '/go/src/github.com/julienschmidt/httprouter'...
fatal: unable to access 'https://github.com/julienschmidt/httprouter/': Could not resolve host: github.com
package github.com/julienschmidt/httprouter: exit status 128
@fcrisciani

This comment has been minimized.

Show comment
Hide comment
@fcrisciani

fcrisciani Jun 18, 2018

Member

@mprobst in case of standalone container, you are in control on which dns to use, if you don't specify anything you will get by default the google ones. if 8.8.8.8 does not work for you, you can specify others with --dns as per https://docs.docker.com/engine/reference/commandline/run

Member

fcrisciani commented Jun 18, 2018

@mprobst in case of standalone container, you are in control on which dns to use, if you don't specify anything you will get by default the google ones. if 8.8.8.8 does not work for you, you can specify others with --dns as per https://docs.docker.com/engine/reference/commandline/run

@fcrisciani fcrisciani closed this Jun 18, 2018

@mprobst

This comment has been minimized.

Show comment
Hide comment
@mprobst

mprobst Jun 19, 2018

@fcrisciani my host system has a /etc/resolv.conf that has a working entry for name resolution. Docker ignores this, and then fails at runtime. That does seem like a bug to me. Can you clarify? It does seem very surprising.

If this is how Docker does and will behave, I guess this is an upstream bug in Ubuntu's package setup.

mprobst commented Jun 19, 2018

@fcrisciani my host system has a /etc/resolv.conf that has a working entry for name resolution. Docker ignores this, and then fails at runtime. That does seem like a bug to me. Can you clarify? It does seem very surprising.

If this is how Docker does and will behave, I guess this is an upstream bug in Ubuntu's package setup.

@fcrisciani

This comment has been minimized.

Show comment
Hide comment
@fcrisciani

fcrisciani Jun 19, 2018

Member

@mprobst
a container spawned with docker run and not networking options will also create a network namespace, isolating it from the host namespace. this will not allow the container to communicate with any of the host resources, in this case the 127.0.0.53 dns.

Member

fcrisciani commented Jun 19, 2018

@mprobst
a container spawned with docker run and not networking options will also create a network namespace, isolating it from the host namespace. this will not allow the container to communicate with any of the host resources, in this case the 127.0.0.53 dns.

@comdw

This comment has been minimized.

Show comment
Hide comment
@comdw

comdw Jun 22, 2018

I agree there is an issue here somewhere...

I have docker 18.03.0-ce running on Ubuntu 16.04 with custom DNS entries in /etc/resolv.conf, e.g.: nameserver 172.99.1.7, and in this environment if I look at /etc/resolv.conf inside a container I see the same thing: nameserver 172.99.1.7

Now on a new machine running docker 18.05.0-ce on Ubuntu 18.04 with the same DNS setup but configured with netplan (which is part of 18.04 changes). The /etc/resolv.conf file should not be edited by hand and uses systemd-resolved - it has an internal nameserver 127.0.0.53 entry in /etc/resolv.conf. Now inside the container I don't see this entry copied but instead get the default google ones (8.8.8.8, 8.8.4.4).

Clearly Ubuntu is different w.r.t. DNS in 18.04, but I haven't confirmed whether Docker is also behaving differently, or just defaults to Google DNS because it somehow knows that 127.0.0.53 is not going to work inside the container.

My workaround for now is to use the --dns option when starting containers to manually specify the hosts, but I would rather the DNS inheritance from the host continued to work!

Documentation ref:
https://docs.docker.com/config/containers/container-networking/#dns-services
"By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis"

Related issue?
#2068

comdw commented Jun 22, 2018

I agree there is an issue here somewhere...

I have docker 18.03.0-ce running on Ubuntu 16.04 with custom DNS entries in /etc/resolv.conf, e.g.: nameserver 172.99.1.7, and in this environment if I look at /etc/resolv.conf inside a container I see the same thing: nameserver 172.99.1.7

Now on a new machine running docker 18.05.0-ce on Ubuntu 18.04 with the same DNS setup but configured with netplan (which is part of 18.04 changes). The /etc/resolv.conf file should not be edited by hand and uses systemd-resolved - it has an internal nameserver 127.0.0.53 entry in /etc/resolv.conf. Now inside the container I don't see this entry copied but instead get the default google ones (8.8.8.8, 8.8.4.4).

Clearly Ubuntu is different w.r.t. DNS in 18.04, but I haven't confirmed whether Docker is also behaving differently, or just defaults to Google DNS because it somehow knows that 127.0.0.53 is not going to work inside the container.

My workaround for now is to use the --dns option when starting containers to manually specify the hosts, but I would rather the DNS inheritance from the host continued to work!

Documentation ref:
https://docs.docker.com/config/containers/container-networking/#dns-services
"By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis"

Related issue?
#2068

@diegoquintanav

This comment has been minimized.

Show comment
Hide comment
@diegoquintanav

diegoquintanav Jul 4, 2018

Probably related moby/moby#36153
At that moment it was with ubuntu 16.04, now with 18.04

If it's worth something, I posted in SO with updated information https://stackoverflow.com/questions/51105875/internet-connection-not-working-networkmanager-not-working-after-installing-dock

diegoquintanav commented Jul 4, 2018

Probably related moby/moby#36153
At that moment it was with ubuntu 16.04, now with 18.04

If it's worth something, I posted in SO with updated information https://stackoverflow.com/questions/51105875/internet-connection-not-working-networkmanager-not-working-after-installing-dock

@ptink

This comment has been minimized.

Show comment
Hide comment
@ptink

ptink Jul 6, 2018

I really don't think this issue should have been closed- docker is broken out of the box on any corporate network, and in a non-obvious way that will usually require the hunting down of this issue in order to find a work around.

Would it not be possible for docker to detect the presence of nameserver 127.0.0.53 in resolv.conf in a similar way to how systemd-resolve does and use /run/systemd/resolv/resolv.conf (which contains the actual DNS nameserver rather than the local) instead?

ptink commented Jul 6, 2018

I really don't think this issue should have been closed- docker is broken out of the box on any corporate network, and in a non-obvious way that will usually require the hunting down of this issue in order to find a work around.

Would it not be possible for docker to detect the presence of nameserver 127.0.0.53 in resolv.conf in a similar way to how systemd-resolve does and use /run/systemd/resolv/resolv.conf (which contains the actual DNS nameserver rather than the local) instead?

@kaneg

This comment has been minimized.

Show comment
Hide comment
@kaneg

kaneg Jul 9, 2018

I also encountered the same issue after upgrading to Ubuntu 18.04.
The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google's public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way.
I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

kaneg commented Jul 9, 2018

I also encountered the same issue after upgrading to Ubuntu 18.04.
The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google's public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way.
I'm not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver:
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

@ctelfer

This comment has been minimized.

Show comment
Hide comment
@ctelfer

ctelfer Jul 11, 2018

Contributor

I looked into this a bit deeper to add some color as to what is going on here. Docker spawns a resolver for all user-defined networks. From moby/moby:

        if !containertypes.NetworkMode(n.Name()).IsUserDefined() {
                createOptions = append(createOptions, libnetwork.CreateOptionDisableResolution())
        }    

and from libnetwork:

func CreateOptionDisableResolution() EndpointOption {
        return func(ep *endpoint) {
                ep.disableResolution = true
        }
}
...
func (ep *endpoint) needResolver() bool {
        ep.Lock()
        defer ep.Unlock()
        return !ep.disableResolution
}
...
func (sb *sandbox) populateNetworkResources(ep *endpoint) error {
...
        if ep.needResolver() {
                sb.startResolver(false)
        }

So for non-user-defined networks (e.g. bridge) docker/libnetwork specifically does not start any sort of proxying resolver. I verified this experimentally by creating a fresh bridge network and running docker run -it --network=mynet --privileged nicolaka/netshoot iptables -L -t nat -n and seeing the IPtables redirects to the proxy resolver.

...
Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:40937
DNAT       udp  --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:57704
...

Running the same command without the --network=mynet (putting the container in the default bridge network) will show an empty set of iptables rules.

Containers without resolver proxying have no way to reach the host's 127.0.0.* network. As @fcrisciani said above, any network traffic directed to such an address will go to the container's network namespace's loopback address which is not plumbed to answer such requests. This is why libnetwork specifically filters all 127.* nameservers from the generated /etc/resolv.conf that goes in each container. Docker engine reads the (configurably-located if I read the code correctly) resolv.conf file for the system, but then filters out those localhost nameservers in the resolv.conf that it generates for the container because it knows the container can't reach them.

This is why @comdw saw the name server's being copied in, but others have not in ubuntu 18.04. A simple workaround if your system is unable to reach 8.8.8.8 / 8.8.4.4 (the defaults provided when no other host resolution seems possible) may well be to place your containers in a user-defined bridge network rather than the default bridge. (Haven't tested this yet on 18.04, though. If someone would that would be helpful.)

Given how annoying this would be in general, the right solution would seem to be a change of policy (or an override) in moby to re-enable proxy resolution for the bridge network. But it doesn't look like something we can fix from libnetwork ... at least not cleanly.

Contributor

ctelfer commented Jul 11, 2018

I looked into this a bit deeper to add some color as to what is going on here. Docker spawns a resolver for all user-defined networks. From moby/moby:

        if !containertypes.NetworkMode(n.Name()).IsUserDefined() {
                createOptions = append(createOptions, libnetwork.CreateOptionDisableResolution())
        }    

and from libnetwork:

func CreateOptionDisableResolution() EndpointOption {
        return func(ep *endpoint) {
                ep.disableResolution = true
        }
}
...
func (ep *endpoint) needResolver() bool {
        ep.Lock()
        defer ep.Unlock()
        return !ep.disableResolution
}
...
func (sb *sandbox) populateNetworkResources(ep *endpoint) error {
...
        if ep.needResolver() {
                sb.startResolver(false)
        }

So for non-user-defined networks (e.g. bridge) docker/libnetwork specifically does not start any sort of proxying resolver. I verified this experimentally by creating a fresh bridge network and running docker run -it --network=mynet --privileged nicolaka/netshoot iptables -L -t nat -n and seeing the IPtables redirects to the proxy resolver.

...
Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:40937
DNAT       udp  --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:57704
...

Running the same command without the --network=mynet (putting the container in the default bridge network) will show an empty set of iptables rules.

Containers without resolver proxying have no way to reach the host's 127.0.0.* network. As @fcrisciani said above, any network traffic directed to such an address will go to the container's network namespace's loopback address which is not plumbed to answer such requests. This is why libnetwork specifically filters all 127.* nameservers from the generated /etc/resolv.conf that goes in each container. Docker engine reads the (configurably-located if I read the code correctly) resolv.conf file for the system, but then filters out those localhost nameservers in the resolv.conf that it generates for the container because it knows the container can't reach them.

This is why @comdw saw the name server's being copied in, but others have not in ubuntu 18.04. A simple workaround if your system is unable to reach 8.8.8.8 / 8.8.4.4 (the defaults provided when no other host resolution seems possible) may well be to place your containers in a user-defined bridge network rather than the default bridge. (Haven't tested this yet on 18.04, though. If someone would that would be helpful.)

Given how annoying this would be in general, the right solution would seem to be a change of policy (or an override) in moby to re-enable proxy resolution for the bridge network. But it doesn't look like something we can fix from libnetwork ... at least not cleanly.

@mushkevych

This comment has been minimized.

Show comment
Hide comment
@mushkevych

mushkevych Aug 15, 2018

As this thread is referenced by Google, let me provide "how-to" on resolving the issue with the docker builds on Ubuntu 18.04.

  • find out the DNS servers with command (your IP addressed will be different):
$ nmcli dev show | grep DNS
IP4.DNS[1]:                             10.11.12.13
IP4.DNS[2]:                             10.11.12.14
  • register these IPs with the /etc/resolv.conf:
sudo -s
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail
  • reboot the OS and check whether the IPs have been registered:
cat /etc/resolv.conf   

mushkevych commented Aug 15, 2018

As this thread is referenced by Google, let me provide "how-to" on resolving the issue with the docker builds on Ubuntu 18.04.

  • find out the DNS servers with command (your IP addressed will be different):
$ nmcli dev show | grep DNS
IP4.DNS[1]:                             10.11.12.13
IP4.DNS[2]:                             10.11.12.14
  • register these IPs with the /etc/resolv.conf:
sudo -s
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail
  • reboot the OS and check whether the IPs have been registered:
cat /etc/resolv.conf   
@bmurphy1976

This comment has been minimized.

Show comment
Hide comment
@bmurphy1976

bmurphy1976 Aug 16, 2018

We solved this by adding the following to every docker run command. Using this approach we at least didn't have to modify any system wide files on our file system.

--dns `cat /run/systemd/resolve/resolv.conf | grep nameserver | head -n 1 | awk '{print $2}'`

The fact that we had to do this on all of our Ubuntu 18.04 servers is obnoxious.

bmurphy1976 commented Aug 16, 2018

We solved this by adding the following to every docker run command. Using this approach we at least didn't have to modify any system wide files on our file system.

--dns `cat /run/systemd/resolve/resolv.conf | grep nameserver | head -n 1 | awk '{print $2}'`

The fact that we had to do this on all of our Ubuntu 18.04 servers is obnoxious.

@diegoquintanav

This comment has been minimized.

Show comment
Hide comment
@diegoquintanav

diegoquintanav Aug 16, 2018

@mushkevych What OS are you using? in 18.04 there's no resolv.conf.d/tail out of the box.

diegoquintanav commented Aug 16, 2018

@mushkevych What OS are you using? in 18.04 there's no resolv.conf.d/tail out of the box.

@mushkevych

This comment has been minimized.

Show comment
Hide comment
@mushkevych

mushkevych Aug 16, 2018

@diegoquintanav I am using Ubuntu 18.04
You are correct that resolv.conf.d/tail is not present out of the box, so the commands above are creating it.

mushkevych commented Aug 16, 2018

@diegoquintanav I am using Ubuntu 18.04
You are correct that resolv.conf.d/tail is not present out of the box, so the commands above are creating it.

@diegoquintanav

This comment has been minimized.

Show comment
Hide comment
@diegoquintanav

diegoquintanav Aug 16, 2018

@mushkevych It didn't solve it for me, do you have perhaps resolvconf installed? It does not come shipped with 18.04 neither.

diegoquintanav commented Aug 16, 2018

@mushkevych It didn't solve it for me, do you have perhaps resolvconf installed? It does not come shipped with 18.04 neither.

@yaleman

This comment has been minimized.

Show comment
Hide comment
@yaleman

yaleman Aug 24, 2018

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

yaleman commented Aug 24, 2018

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

@carletes

This comment has been minimized.

Show comment
Hide comment
@carletes

carletes Aug 24, 2018

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

I followed @kaneg's approach:

$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 32 Aug  6 12:58 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
$

According to the man page systemd-resolved(8) on my Ubuntu 18.04 system, this is a clean and supported approach (see section "/ETC/RESOLV.CONF" there).

With this setup in place, name resolution inside containers works just fine for me.

carletes commented Aug 24, 2018

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I'm looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

I followed @kaneg's approach:

$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 32 Aug  6 12:58 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
$

According to the man page systemd-resolved(8) on my Ubuntu 18.04 system, this is a clean and supported approach (see section "/ETC/RESOLV.CONF" there).

With this setup in place, name resolution inside containers works just fine for me.

@yaleman

This comment has been minimized.

Show comment
Hide comment
@yaleman

yaleman Aug 24, 2018

Thanks @carletes and @kaneg that works perfectly in my testing.

yaleman commented Aug 24, 2018

Thanks @carletes and @kaneg that works perfectly in my testing.

@glaux

This comment has been minimized.

Show comment
Hide comment
@glaux

glaux Aug 28, 2018

I definitely agree that this should be treated like a bug. The solution @kaneg suggests unfortunately doesn't work on my system (Ubuntu 18.04, Docker 18.06.0-ce). This is my output:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 32 Aug 28 10:06 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

however the container still have the incorrect configuration when started with docker-compose.

The only 'workaround' I've found is to manually append the contents of /etc/resolv.conf from the host to the same file in the container. As others have pointed out, ADD or COPY on build doesn't work. Perhaps the file could be populated with the correct dns info with an entrypoint script, but honestly I've already wasted a few hours trying to get this to work and I don't restart my containers too often.

glaux commented Aug 28, 2018

I definitely agree that this should be treated like a bug. The solution @kaneg suggests unfortunately doesn't work on my system (Ubuntu 18.04, Docker 18.06.0-ce). This is my output:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 32 Aug 28 10:06 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

however the container still have the incorrect configuration when started with docker-compose.

The only 'workaround' I've found is to manually append the contents of /etc/resolv.conf from the host to the same file in the container. As others have pointed out, ADD or COPY on build doesn't work. Perhaps the file could be populated with the correct dns info with an entrypoint script, but honestly I've already wasted a few hours trying to get this to work and I don't restart my containers too often.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment