New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker should use the host network DNS server #23910

Closed
nottrobin opened this Issue Jun 23, 2016 · 48 comments

Comments

Projects
None yet
@nottrobin

nottrobin commented Jun 23, 2016

Summary

In networks where external DNS servers are blocked, Docker containers running on Ubuntu hosts can't resolve DNS at all because they are trying to use 8.8.8.8 as their DNS server. Docker should detect the network DNS server.

Detail

When spinning up a container, Docker will by default check for a DNS server defined in /etc/resolv.conf in the host OS, and if it doesn't find one, or finds only 127.0.0.1, will opt to use Google's public DNS server 8.8.8.8.

My development machine is running Ubuntu 16.04 which uses dnsmasq by default, so /etc/resolv.conf is always set to 127.0.0.1, even though it is usually actually getting its DNS settings from whatever network its connected to:

$ cat /etc/resolv.conf | grep nameserver  # What Docker sees
nameserver 127.0.1.1
$ nmcli dev show | grep IP4.DNS  # My actual DNS server
IP4.DNS[1]:                             10.1.1.3

and so Docker containers always default to using 8.8.8.8 rather than using the same DNS server as the host OS.

In my office network external DNS servers are blocked, and so me and my whole team are finding docker containers failing with obscure errors which result from failing to resolve DNS.

There is a workaround, which I help walk all my team members through so they can still use Docker. Although it's frustrating that whenever Docker gets updated, it seems the daemon config file is overwritten and we have to implement the fix all over again. This is less of a problem for me, but is causing significant trouble for our less systems-focused developers.

Is there any hope that Docker might be able to intelligently pick up the network's DNS server in the future?

Steps to reproduce the issue

  1. Use a host OS that makes use of dnsmasq (e.g. Ubuntu since 12.04)
  2. Connect to a network that blocks access to external DNS servers like 8.8.8.8
  3. Try to resolve DNS with Docker (docker run busybox nslookup google.com)

What I get

$ docker run busybox nslookup google.com
Server:    8.8.8.8
Address 1: 8.8.8.8

nslookup: can't resolve 'google.com'

What I expected

$ docker run busybox nslookup google.com
Server:    10.1.1.3
Address 1: 10.1.1.3

Name:      google.com
Address 1: 2a00:1450:4009:811::200e lhr26s02-in-x200e.1e100.net
Address 2: 216.58.198.174 lhr25s10-in-f14.1e100.net

System information

I'm running Xenial natively on a Dell XPS 13 9350.

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

$ docker info
Containers: 61
 Running: 3
 Paused: 0
 Stopped: 58
Images: 421
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 441
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-24-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.54 GiB
Name: xps
ID: G4MN:GTXD:4KZP:PTZC:DBYK:WOLA:R3GF:TKLW:ZOOX:NXZT:ALNG:F22D
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: nottrobin
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire
Contributor

nathanleclaire commented Jun 23, 2016

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 24, 2016

Contributor

The systemd unit file should not be edited directly.
Systemd has a facility called "drop-ins" that allow you to override/augment the unit file.
https://docs.docker.com/engine/admin/systemd/

In addition, it would probably be best to configure the docker daemon via it's config file rather than the systemd unit file. By default docker reads config from /etc/docker/daemon.json. https://docs.docker.com/engine/reference/commandline/dockerd/#linux-configuration-file

Contributor

cpuguy83 commented Jun 24, 2016

The systemd unit file should not be edited directly.
Systemd has a facility called "drop-ins" that allow you to override/augment the unit file.
https://docs.docker.com/engine/admin/systemd/

In addition, it would probably be best to configure the docker daemon via it's config file rather than the systemd unit file. By default docker reads config from /etc/docker/daemon.json. https://docs.docker.com/engine/reference/commandline/dockerd/#linux-configuration-file

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin Jun 24, 2016

Ah! Wonderful suggestions. Thanks @cpuguy83, I'll update my blog post.

nottrobin commented Jun 24, 2016

Ah! Wonderful suggestions. Thanks @cpuguy83, I'll update my blog post.

@ovidiub13

This comment has been minimized.

Show comment
Hide comment
@ovidiub13

ovidiub13 Jun 30, 2016

I've solved this by adding a daemon configuration file. But I would sure love for this to be implemented.

ovidiub13 commented Jun 30, 2016

I've solved this by adding a daemon configuration file. But I would sure love for this to be implemented.

@iflowfor8hours

This comment has been minimized.

Show comment
Hide comment
@iflowfor8hours

iflowfor8hours Aug 20, 2016

@cpuguy83 Nice one, I didn't know about the /etc/docker/daemon.json. Might that file get created in a default install to signal to users this is where configuration should live? I went straight for the systemd drop-ins.

iflowfor8hours commented Aug 20, 2016

@cpuguy83 Nice one, I didn't know about the /etc/docker/daemon.json. Might that file get created in a default install to signal to users this is where configuration should live? I went straight for the systemd drop-ins.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Aug 20, 2016

Contributor

It is not very easy to implement this as docker has no way to know the
upstream servers. If you set your local DNS relay on localhost to also
listen on the docker0 IP and put that address in resolv.conf that would
also work.

On 30 Jun 2016 3:12 p.m., "Ovidiu-Florin BOGDAN" notifications@github.com
wrote:

I've solved this by adding a daemon configuration file. But I would sure
love for this to be implemented.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#23910 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAdcPFlthUzL2cze4iV_wGe0xRFhQPRpks5qQ87WgaJpZM4I9Lm8
.

Contributor

justincormack commented Aug 20, 2016

It is not very easy to implement this as docker has no way to know the
upstream servers. If you set your local DNS relay on localhost to also
listen on the docker0 IP and put that address in resolv.conf that would
also work.

On 30 Jun 2016 3:12 p.m., "Ovidiu-Florin BOGDAN" notifications@github.com
wrote:

I've solved this by adding a daemon configuration file. But I would sure
love for this to be implemented.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#23910 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAdcPFlthUzL2cze4iV_wGe0xRFhQPRpks5qQ87WgaJpZM4I9Lm8
.

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin Sep 9, 2016

@justincormack I can discover the DNS servers by doing nmcli dev show | grep DNS. Couldn't docker do something similar? I mean, more generally, if the system knows its DNS servers (which it must) then surely docker can discover them somehow.

nottrobin commented Sep 9, 2016

@justincormack I can discover the DNS servers by doing nmcli dev show | grep DNS. Couldn't docker do something similar? I mean, more generally, if the system knows its DNS servers (which it must) then surely docker can discover them somehow.

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Sep 19, 2016

On Ubuntu, I settled on doing this ( as @justincormack describes) by...

  • Adding a /etc/docker/daemon.json that passed in the docker0 bridge IP as a DNS server
  • Adding a /etc/NetworkManager/dnsmasq.d/docker-bridge.conf file that directs the NM-managed DNS cache to listen to the docker0 bridge IP as well as 127.0.1.1

Now container DNS lookups go through your local dnsmasq instance, so DNS in the container should work just as well as DNS on the host.

@nottrobin - my problem specifically was that my system did know it's DNS servers and was passing them to the container - but because those servers were on a VPN, there was no route to them from inside the container, hence all DNS lookups failed. This was partly down to the non-NetworkManager based VPN client we're using that overwites resolv.conf. Turned that "feature" off and instead added config for that subdomain on NetworkManager's dnsmasq itself.

awilkins commented Sep 19, 2016

On Ubuntu, I settled on doing this ( as @justincormack describes) by...

  • Adding a /etc/docker/daemon.json that passed in the docker0 bridge IP as a DNS server
  • Adding a /etc/NetworkManager/dnsmasq.d/docker-bridge.conf file that directs the NM-managed DNS cache to listen to the docker0 bridge IP as well as 127.0.1.1

Now container DNS lookups go through your local dnsmasq instance, so DNS in the container should work just as well as DNS on the host.

@nottrobin - my problem specifically was that my system did know it's DNS servers and was passing them to the container - but because those servers were on a VPN, there was no route to them from inside the container, hence all DNS lookups failed. This was partly down to the non-NetworkManager based VPN client we're using that overwites resolv.conf. Turned that "feature" off and instead added config for that subdomain on NetworkManager's dnsmasq itself.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Sep 20, 2016

Contributor

@nottrobin nmcli only applies to systems running NetworkManager, there is not an equivalent for other setups.

Contributor

justincormack commented Sep 20, 2016

@nottrobin nmcli only applies to systems running NetworkManager, there is not an equivalent for other setups.

@iraadit

This comment has been minimized.

Show comment
Hide comment
@iraadit

iraadit Oct 10, 2016

Thank you A LOT for the workaround @nottrobin, working great even when doing "docker build"
It had been three days of looking for a solution without success before finding your post

iraadit commented Oct 10, 2016

Thank you A LOT for the workaround @nottrobin, working great even when doing "docker build"
It had been three days of looking for a solution without success before finding your post

@bsingr

This comment has been minimized.

Show comment
Hide comment
@bsingr

bsingr Nov 23, 2016

Your solution works @awilkins.

Additionally it requires to allow port 53 traffic on the ubuntu firewall for the docker0 bridge ip.

Quick test: disable the ubuntu firewall completely with sudo ufw disable

bsingr commented Nov 23, 2016

Your solution works @awilkins.

Additionally it requires to allow port 53 traffic on the ubuntu firewall for the docker0 bridge ip.

Quick test: disable the ubuntu firewall completely with sudo ufw disable

@krak3n

This comment has been minimized.

Show comment
Hide comment
@krak3n

krak3n Dec 1, 2016

@bsingr what rule did you use? I've tried loads and can't get it to work on Ubuntu 16.04 - i've resorted to host network mode 😭

krak3n commented Dec 1, 2016

@bsingr what rule did you use? I've tried loads and can't get it to work on Ubuntu 16.04 - i've resorted to host network mode 😭

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Dec 19, 2016

@nottrobin Instead of trying to detect the external resolver used by the host resolver the better approach is to allow the containers' DNS queries to be forwarded to the host resolver if the resolv.conf has a loopback address. Using the host resolver has other use cases as well. Its doable with the docker embedded DNS server which is used by default for user defined networks. I pushed a PR in libnetwork for this.

sanimej commented Dec 19, 2016

@nottrobin Instead of trying to detect the external resolver used by the host resolver the better approach is to allow the containers' DNS queries to be forwarded to the host resolver if the resolv.conf has a loopback address. Using the host resolver has other use cases as well. Its doable with the docker embedded DNS server which is used by default for user defined networks. I pushed a PR in libnetwork for this.

@butlermd

This comment has been minimized.

Show comment
Hide comment
@butlermd

butlermd Jan 9, 2017

@awilkins or @bsingr, do either of you have more details about how you configured docker-bridge.conf? The daemon.json file seems like it should be simple enough:

{
  "dns": [
    "172.17.0.1"
  ]
}

But I'm unclear on how you configured dnsmasq.

butlermd commented Jan 9, 2017

@awilkins or @bsingr, do either of you have more details about how you configured docker-bridge.conf? The daemon.json file seems like it should be simple enough:

{
  "dns": [
    "172.17.0.1"
  ]
}

But I'm unclear on how you configured dnsmasq.

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Jan 9, 2017

On ubuntu, add a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf

listen-address=172.17.0.1

By default it only listens to DNS requests from 127.0.0.1 (ie, your computer). This tells it to listen to the docker bridge also.

awilkins commented Jan 9, 2017

On ubuntu, add a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf

listen-address=172.17.0.1

By default it only listens to DNS requests from 127.0.0.1 (ie, your computer). This tells it to listen to the docker bridge also.

@butlermd

This comment has been minimized.

Show comment
Hide comment
@butlermd

butlermd Jan 9, 2017

@awilkins Thanks for the clarification!

butlermd commented Jan 9, 2017

@awilkins Thanks for the clarification!

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Jan 9, 2017

Also looks like a PR with support for doing this less cryptically landed a few days ago in libnetwork

AFAIK though this won't work for the docker0 network bridge.

docker/libnetwork@1ed42d1

awilkins commented Jan 9, 2017

Also looks like a PR with support for doing this less cryptically landed a few days ago in libnetwork

AFAIK though this won't work for the docker0 network bridge.

docker/libnetwork@1ed42d1

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Jan 9, 2017

@awilkins Yes, it doesn't work for docker0 bridge because the Docker DNS server is used only for the user created networks. The fix to forward the queries to host loopback resolver relies on the embedded DNS server.

sanimej commented Jan 9, 2017

@awilkins Yes, it doesn't work for docker0 bridge because the Docker DNS server is used only for the user created networks. The fix to forward the queries to host loopback resolver relies on the embedded DNS server.

@thaJeztah thaJeztah closed this in #29891 Jan 11, 2017

@thaJeztah thaJeztah added this to the 1.14.0 milestone Jan 11, 2017

@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Jan 16, 2017

Contributor

@sanimej just to make sure i understand the patch correctly, it only allows docker embedded DNS server to forward queries to the local nameserver ?

We are using https://github.com/gliderlabs/hostlocal, which will still be required if we dont want to use docker embedded DNS server, is that correct ?

Contributor

dqminh commented Jan 16, 2017

@sanimej just to make sure i understand the patch correctly, it only allows docker embedded DNS server to forward queries to the local nameserver ?

We are using https://github.com/gliderlabs/hostlocal, which will still be required if we dont want to use docker embedded DNS server, is that correct ?

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Jan 17, 2017

@dqminh Yes, this patch essentially differentiates between loopback IP in the container namespace vs host namespace. This allows the embedded DNS server to forward the queries appropriately. From a quick glance it looks like hostlocal package is intended to be used with containers in net=host mode. This patch is not relevant in that case.

sanimej commented Jan 17, 2017

@dqminh Yes, this patch essentially differentiates between loopback IP in the container namespace vs host namespace. This allows the embedded DNS server to forward the queries appropriately. From a quick glance it looks like hostlocal package is intended to be used with containers in net=host mode. This patch is not relevant in that case.

@dqminh

This comment has been minimized.

Show comment
Hide comment
@dqminh

dqminh Jan 17, 2017

Contributor

@sanimej i think hostlocal should work in bridge mode as well since it adds a route in the host. Thanks for the confimation about the embedded DNS server.

Contributor

dqminh commented Jan 17, 2017

@sanimej i think hostlocal should work in bridge mode as well since it adds a route in the host. Thanks for the confimation about the embedded DNS server.

@setiseta

This comment has been minimized.

Show comment
Hide comment
@setiseta

setiseta Jan 30, 2017

hm, just to ask, is it possible to setup a dnsmasq in a container as default dns for other dockers in any way?
i just cant find a description about this on the internet.
is this not possible?

setiseta commented Jan 30, 2017

hm, just to ask, is it possible to setup a dnsmasq in a container as default dns for other dockers in any way?
i just cant find a description about this on the internet.
is this not possible?

@webdobe

This comment has been minimized.

Show comment
Hide comment
@webdobe

webdobe Jun 14, 2017

I ended up resolving this by adding the following to my php container:

extra_hosts: - "myhost.whatever.tld:172.17.0.1"

Since it is my dev this was an easy enough fix.

webdobe commented Jun 14, 2017

I ended up resolving this by adding the following to my php container:

extra_hosts: - "myhost.whatever.tld:172.17.0.1"

Since it is my dev this was an easy enough fix.

@xenithorb

This comment has been minimized.

Show comment
Hide comment

xenithorb commented Jun 14, 2017

@alu

This comment has been minimized.

Show comment
Hide comment
@alu

alu Aug 29, 2017

@sanimej

Hi. I have a question that is How resolve 127.0.0.1 in container.
I read around here and my answer is below.

  1. If --dns option apply when docker run or stack deploy, 127.0.0.1 refer to container self.
  2. if /etc/resolv.conf includes 127.0.0.x nameserver and stack deply, 127.0.0.1 refer to host that is running it.

Is it right?

Thanks.

alu commented Aug 29, 2017

@sanimej

Hi. I have a question that is How resolve 127.0.0.1 in container.
I read around here and my answer is below.

  1. If --dns option apply when docker run or stack deploy, 127.0.0.1 refer to container self.
  2. if /etc/resolv.conf includes 127.0.0.x nameserver and stack deply, 127.0.0.1 refer to host that is running it.

Is it right?

Thanks.

@zanaca

This comment has been minimized.

Show comment
Hide comment
@zanaca

zanaca Dec 22, 2017

Guys, for that "problem" I have made docker-dns.

zanaca commented Dec 22, 2017

Guys, for that "problem" I have made docker-dns.

@mimousewu

This comment has been minimized.

Show comment
Hide comment
@mimousewu

mimousewu Jan 25, 2018

@zanaca Realy Thanks! This DNS issue last for years. I don't know why the official guys seem bland for this. And even you can not find any detail about how it implements.

mimousewu commented Jan 25, 2018

@zanaca Realy Thanks! This DNS issue last for years. I don't know why the official guys seem bland for this. And even you can not find any detail about how it implements.

@akravetz

This comment has been minimized.

Show comment
Hide comment
@akravetz

akravetz Feb 26, 2018

+1 surprised this is still an issue. Really complicates our dev process using ubuntu on vmware.

akravetz commented Feb 26, 2018

+1 surprised this is still an issue. Really complicates our dev process using ubuntu on vmware.

@nileio

This comment has been minimized.

Show comment
Hide comment
@nileio

nileio Mar 9, 2018

+1 still an issue and it is annoying!!

nileio commented Mar 9, 2018

+1 still an issue and it is annoying!!

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin Mar 12, 2018

Yes I'm not quite sure why this is closed. A new team member in our team just installed docker fresh and ran into this problem all over again, and had to follow my blog post to fix it. And I'm frequently getting pings on Twitter because random strangers have similarly had to use my blog post to get Docker working.

I am quite amazed that over a year and a half later it hasn't been deemed important enough to find a proper fix for Ubuntu users (though I'm sure it's a tough problem).

nottrobin commented Mar 12, 2018

Yes I'm not quite sure why this is closed. A new team member in our team just installed docker fresh and ran into this problem all over again, and had to follow my blog post to fix it. And I'm frequently getting pings on Twitter because random strangers have similarly had to use my blog post to get Docker working.

I am quite amazed that over a year and a half later it hasn't been deemed important enough to find a proper fix for Ubuntu users (though I'm sure it's a tough problem).

@ericparton

This comment has been minimized.

Show comment
Hide comment
@ericparton

ericparton Mar 24, 2018

@nottrobin I'm still running into this problem as well, but oddly enough it only seems to affect containers on user defined bridge networks.

So this command will succeed:
docker run busybox nslookup google.com

..but this command won't:
docker run --network my_bridge_network busybox nslookup google.com

I do have other name servers in the resolv.conf in addition to the loopback address, but I don't know if that's a factor.

ericparton commented Mar 24, 2018

@nottrobin I'm still running into this problem as well, but oddly enough it only seems to affect containers on user defined bridge networks.

So this command will succeed:
docker run busybox nslookup google.com

..but this command won't:
docker run --network my_bridge_network busybox nslookup google.com

I do have other name servers in the resolv.conf in addition to the loopback address, but I don't know if that's a factor.

@moy

This comment has been minimized.

Show comment
Hide comment
@moy

moy Apr 24, 2018

I tried summarizing the possible solutions here: https://stackoverflow.com/questions/49998099/dns-not-working-within-docker-containers-when-host-uses-dnsmasq-and-googles-dns

Feel free to add or edit answers there.

moy commented Apr 24, 2018

I tried summarizing the possible solutions here: https://stackoverflow.com/questions/49998099/dns-not-working-within-docker-containers-when-host-uses-dnsmasq-and-googles-dns

Feel free to add or edit answers there.

@dashesy

This comment has been minimized.

Show comment
Hide comment
@dashesy

dashesy May 2, 2018

@moy your answer here is the cleanest way to handle this. I wonder why installing docker does not just drop a /etc/NetworkManager/dnsmasq.d/docker-bridge.conf and fix daemon.json to fix the problem for all.

dashesy commented May 2, 2018

@moy your answer here is the cleanest way to handle this. I wonder why installing docker does not just drop a /etc/NetworkManager/dnsmasq.d/docker-bridge.conf and fix daemon.json to fix the problem for all.

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin May 8, 2018

@moy that answer is brilliant. I'll try it out, confirm it works, and then update my blog post to mention it.

This (implementing /etc/NetworkManager/dnsmasq.d/docker-bridge.conf and configuring Docker to use 172.17.0.1 as its first DNS server) is a solution that Docker could very easily implement in their Ubuntu package, providing a neat fix for this issue. If any of the project maintainers see this, maybe you'd consider reopening this issue and starting work on the fix?

nottrobin commented May 8, 2018

@moy that answer is brilliant. I'll try it out, confirm it works, and then update my blog post to mention it.

This (implementing /etc/NetworkManager/dnsmasq.d/docker-bridge.conf and configuring Docker to use 172.17.0.1 as its first DNS server) is a solution that Docker could very easily implement in their Ubuntu package, providing a neat fix for this issue. If any of the project maintainers see this, maybe you'd consider reopening this issue and starting work on the fix?

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin May 18, 2018

@moy that did not work at all for me or my boss, both on Ubuntu 18.04. Shame, I was excited about it.

nottrobin commented May 18, 2018

@moy that did not work at all for me or my boss, both on Ubuntu 18.04. Shame, I was excited about it.

@moy

This comment has been minimized.

Show comment
Hide comment
@moy

moy May 18, 2018

@nottrobin : any detail on what did not work? I tested my answer on a Ubuntu 16.04 virtual machine (to start from a clean install) and on my laptop (was still Ubuntu 16.04 at the time, but I can re-test with 18.04 when I get it into a network where public DNS are filtered).

moy commented May 18, 2018

@nottrobin : any detail on what did not work? I tested my answer on a Ubuntu 16.04 virtual machine (to start from a clean install) and on my laptop (was still Ubuntu 16.04 at the time, but I can re-test with 18.04 when I get it into a network where public DNS are filtered).

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin May 18, 2018

@moy I'm afraid I don't have very deep knowledge of what didn't work, I don't know network-manager or dnsmasq very well. All I can tell you is the steps I took:

  • Checked ip addr to confirm that docker0 mentions inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
  • Added listen-address=172.17.0.1 to /etc/NetworkManager/dnsmasq.d/docker-bridge.conf
  • sudo service network-manager restart
  • Tried sudo docker run -ti --dns 172.17.0.1 mmoy/ubuntu-netutils ping www.example.com, which couldn't find the host
  • Added {"dns": ["172.17.0.1"]} to /etc/docker/daemon.json
  • sudo service docker restart
  • sudo docker run -ti mmoy/ubuntu-netutils ping www.example.com still couldn't find the host
  • I also tried the method of host lookups used in my blog post: docker run --dns 172.17.0.1 busybox nslookup google.com or docker run busybox nslookup google.com, they didn't work either

If you let me know something I can do to debug this, I'll try to find time to look into it further.

nottrobin commented May 18, 2018

@moy I'm afraid I don't have very deep knowledge of what didn't work, I don't know network-manager or dnsmasq very well. All I can tell you is the steps I took:

  • Checked ip addr to confirm that docker0 mentions inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
  • Added listen-address=172.17.0.1 to /etc/NetworkManager/dnsmasq.d/docker-bridge.conf
  • sudo service network-manager restart
  • Tried sudo docker run -ti --dns 172.17.0.1 mmoy/ubuntu-netutils ping www.example.com, which couldn't find the host
  • Added {"dns": ["172.17.0.1"]} to /etc/docker/daemon.json
  • sudo service docker restart
  • sudo docker run -ti mmoy/ubuntu-netutils ping www.example.com still couldn't find the host
  • I also tried the method of host lookups used in my blog post: docker run --dns 172.17.0.1 busybox nslookup google.com or docker run busybox nslookup google.com, they didn't work either

If you let me know something I can do to debug this, I'll try to find time to look into it further.

@moy

This comment has been minimized.

Show comment
Hide comment
@moy

moy May 18, 2018

Make sure dnsmasq is actually running on your machine (it wasn't on my Ubuntu 18.04, not sure whether I removed it or the upgrade did).

From the host, try:

host www.google.com 127.0.0.1

This should work, and should force querying the dnsmasq server.

moy commented May 18, 2018

Make sure dnsmasq is actually running on your machine (it wasn't on my Ubuntu 18.04, not sure whether I removed it or the upgrade did).

From the host, try:

host www.google.com 127.0.0.1

This should work, and should force querying the dnsmasq server.

@nottrobin

This comment has been minimized.

Show comment
Hide comment
@nottrobin

nottrobin May 21, 2018

@moy I think it's running, but on 127.0.0.53:

Command-line snippet
$ host www.google.com 127.0.0.1
;; connection timed out; no servers could be reached
$ tail -n 1 /etc/resolv.conf 
nameserver 127.0.0.53
$ host www.google.com 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53
Aliases: 

www.google.com has address 209.85.202.103
www.google.com has address 209.85.202.106
www.google.com has address 209.85.202.99
www.google.com has address 209.85.202.105
www.google.com has address 209.85.202.104
www.google.com has address 209.85.202.147
www.google.com has IPv6 address 2a00:1450:400b:c00::6a

Should this make any difference to the solution?

nottrobin commented May 21, 2018

@moy I think it's running, but on 127.0.0.53:

Command-line snippet
$ host www.google.com 127.0.0.1
;; connection timed out; no servers could be reached
$ tail -n 1 /etc/resolv.conf 
nameserver 127.0.0.53
$ host www.google.com 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53
Aliases: 

www.google.com has address 209.85.202.103
www.google.com has address 209.85.202.106
www.google.com has address 209.85.202.99
www.google.com has address 209.85.202.105
www.google.com has address 209.85.202.104
www.google.com has address 209.85.202.147
www.google.com has IPv6 address 2a00:1450:400b:c00::6a

Should this make any difference to the solution?

@moy

This comment has been minimized.

Show comment
Hide comment
@moy

moy May 21, 2018

Check that it's actually dnsmasq. Google suggests that this IP is used by systemd-resolved, and systemd-resolved was installed on my ubuntu 18.04. If this is the case, then my instructions will have to be adapted to systemd.

moy commented May 21, 2018

Check that it's actually dnsmasq. Google suggests that this IP is used by systemd-resolved, and systemd-resolved was installed on my ubuntu 18.04. If this is the case, then my instructions will have to be adapted to systemd.

@jerrac

This comment has been minimized.

Show comment
Hide comment
@jerrac

jerrac Jul 24, 2018

I just spent a long time trying to figure out why my containers couldn't resolve any domains. This issue is the closest I found to an answer.

I'm on Ubuntu 18.04. Docker version 18.06.0-ce, build 0ffa825.

Ubuntu is not using dnsmasq, at least I think. Judging from the comments in this issue and the SO post my system is using systemd-resolved.

My host can find google.com, etc, just fine.

If I run a container, the container cannot resolve any domains unless I use --network host.

I double checked that my docker daemon service file was told to load the daemon.json file. Note, I have stopped docker from managing the firewall.

I also flushed iptables just to make sure the firewall wasn't in the way.

I've modified my daemon.json file so that it contains my workplaces dns ip's, as well as 1.1.1.1 and 8.8.8.8.

I tried disabling /lib/systemd/system/resolveconf.service and /lib/systemd/system/systemd-resolved.service using systemctl. That ended up breaking dns for my whole system.

I tried creating a custom bridge network and using that when running my containers, the container was still unable to resolve domain names.

Is there anything else I can try?

jerrac commented Jul 24, 2018

I just spent a long time trying to figure out why my containers couldn't resolve any domains. This issue is the closest I found to an answer.

I'm on Ubuntu 18.04. Docker version 18.06.0-ce, build 0ffa825.

Ubuntu is not using dnsmasq, at least I think. Judging from the comments in this issue and the SO post my system is using systemd-resolved.

My host can find google.com, etc, just fine.

If I run a container, the container cannot resolve any domains unless I use --network host.

I double checked that my docker daemon service file was told to load the daemon.json file. Note, I have stopped docker from managing the firewall.

I also flushed iptables just to make sure the firewall wasn't in the way.

I've modified my daemon.json file so that it contains my workplaces dns ip's, as well as 1.1.1.1 and 8.8.8.8.

I tried disabling /lib/systemd/system/resolveconf.service and /lib/systemd/system/systemd-resolved.service using systemctl. That ended up breaking dns for my whole system.

I tried creating a custom bridge network and using that when running my containers, the container was still unable to resolve domain names.

Is there anything else I can try?

@denji

This comment has been minimized.

Show comment
Hide comment
@denji

denji Jul 25, 2018

@jerrac resolve via ipv6 or ipv4? moby/moby#15086

denji commented Jul 25, 2018

@jerrac resolve via ipv6 or ipv4? moby/moby#15086

@jerrac

This comment has been minimized.

Show comment
Hide comment
@jerrac

jerrac Jul 25, 2018

@denji My workplace network is ipv4 only. My host is not configured to use ipv6 on any network. (It's a laptop.)

jerrac commented Jul 25, 2018

@denji My workplace network is ipv4 only. My host is not configured to use ipv6 on any network. (It's a laptop.)

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Jul 25, 2018

As someone else who's run into this : (see my answer on askubuntu...

Ubuntu 18.04 exacerbates this by replacing the NetworkManager managed instance of dnsmasq with systemd-resolved.

systemd-resolved doesn't listen to the docker0 network bridge adapter - and can't be configured to do so, it will only listen on it's hardcoded loopback address.

I worked around this on Ubuntu 18.04 by

  • Disabling the local UDP server on systemd-resolved
  • Installing the NetworkManager dnsmasq support again
  • Reconfiguring it as described in my answer but also
  • Reconfiguring it's primary listen address to 127.0.0.53 (because something in systemd still writes this to /etc/resolvconf and I couldn't work out how to disable that)

awilkins commented Jul 25, 2018

As someone else who's run into this : (see my answer on askubuntu...

Ubuntu 18.04 exacerbates this by replacing the NetworkManager managed instance of dnsmasq with systemd-resolved.

systemd-resolved doesn't listen to the docker0 network bridge adapter - and can't be configured to do so, it will only listen on it's hardcoded loopback address.

I worked around this on Ubuntu 18.04 by

  • Disabling the local UDP server on systemd-resolved
  • Installing the NetworkManager dnsmasq support again
  • Reconfiguring it as described in my answer but also
  • Reconfiguring it's primary listen address to 127.0.0.53 (because something in systemd still writes this to /etc/resolvconf and I couldn't work out how to disable that)
@dashesy

This comment has been minimized.

Show comment
Hide comment
@dashesy

dashesy Jul 25, 2018

@awilkins this sounds like a bug, in systemd. Have you tried opening a bug with them? sure there are more people who use docker and want to try Ubuntu 18 host!

dashesy commented Jul 25, 2018

@awilkins this sounds like a bug, in systemd. Have you tried opening a bug with them? sure there are more people who use docker and want to try Ubuntu 18 host!

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Jul 25, 2018

It appears to be a design choice, rather than a bug :

systemd-resolved deliberately only binds to the loopback adapter

This seems to work properly now ... if you don't use the default network. Have just tested this and using a non-default network results in /etc/resolv.conf in the container using 127.0.0.11 - which is docker's embedded DNS resolver. Docker doesn't do this in the default network because of backward-compatibility.

While this is a PITA it's work-around-able, by specifying a non-default network. Mostly I use docker-compose so with user-defined networks defined in my compose files so this will happen by default. docker build and other ad-hoc docker commands accept --net params.

If you want the total convenience of it Just Working™, there's still the option of turning off that systemd-resolved stub and installing dnsmasq (using the NetworkManager one is still easiest for me on 18.04)


The main gotcha with the above - docker-compose runs containers in a user-defined network, but seems to build them in the default network, so docker-compose up will still fail for the above reasons if it needs to build an image that needs DNS requests to build.

awilkins commented Jul 25, 2018

It appears to be a design choice, rather than a bug :

systemd-resolved deliberately only binds to the loopback adapter

This seems to work properly now ... if you don't use the default network. Have just tested this and using a non-default network results in /etc/resolv.conf in the container using 127.0.0.11 - which is docker's embedded DNS resolver. Docker doesn't do this in the default network because of backward-compatibility.

While this is a PITA it's work-around-able, by specifying a non-default network. Mostly I use docker-compose so with user-defined networks defined in my compose files so this will happen by default. docker build and other ad-hoc docker commands accept --net params.

If you want the total convenience of it Just Working™, there's still the option of turning off that systemd-resolved stub and installing dnsmasq (using the NetworkManager one is still easiest for me on 18.04)


The main gotcha with the above - docker-compose runs containers in a user-defined network, but seems to build them in the default network, so docker-compose up will still fail for the above reasons if it needs to build an image that needs DNS requests to build.

@jerrac

This comment has been minimized.

Show comment
Hide comment
@jerrac

jerrac Jul 25, 2018

@awilkins I did try docker network create ... and then using that custom bridge network when I used docker run. I still ran into the dns issues.

Oddly, if I try the same thing using docker-compose, and a named network, the dns issues go away.

So, what's the different between a docker-compose named network and a docker network create network?

jerrac commented Jul 25, 2018

@awilkins I did try docker network create ... and then using that custom bridge network when I used docker run. I still ran into the dns issues.

Oddly, if I try the same thing using docker-compose, and a named network, the dns issues go away.

So, what's the different between a docker-compose named network and a docker network create network?

@awilkins

This comment has been minimized.

Show comment
Hide comment
@awilkins

awilkins Jul 26, 2018

That seems to work fine for me. Is the problem docker's excruciating pickiness about argument order?

$ docker network create cheese
db2e8491afea95700120a3615d01e7507c45cfe2b34397641eb286a1de4ed44c
$ docker run --net cheese bash cat /etc/resolv.conf
nameserver 127.0.0.11
$ docker run  bash cat /etc/resolv.conf
nameserver 172.17.0.1

awilkins commented Jul 26, 2018

That seems to work fine for me. Is the problem docker's excruciating pickiness about argument order?

$ docker network create cheese
db2e8491afea95700120a3615d01e7507c45cfe2b34397641eb286a1de4ed44c
$ docker run --net cheese bash cat /etc/resolv.conf
nameserver 127.0.0.11
$ docker run  bash cat /etc/resolv.conf
nameserver 172.17.0.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment