Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker should use the host network DNS server #23910

Closed
nottrobin opened this issue Jun 23, 2016 · 63 comments
Closed

Docker should use the host network DNS server #23910

nottrobin opened this issue Jun 23, 2016 · 63 comments

Comments

@nottrobin
Copy link

@nottrobin nottrobin commented Jun 23, 2016

Summary

In networks where external DNS servers are blocked, Docker containers running on Ubuntu hosts can't resolve DNS at all because they are trying to use 8.8.8.8 as their DNS server. Docker should detect the network DNS server.

Detail

When spinning up a container, Docker will by default check for a DNS server defined in /etc/resolv.conf in the host OS, and if it doesn't find one, or finds only 127.0.0.1, will opt to use Google's public DNS server 8.8.8.8.

My development machine is running Ubuntu 16.04 which uses dnsmasq by default, so /etc/resolv.conf is always set to 127.0.0.1, even though it is usually actually getting its DNS settings from whatever network its connected to:

$ cat /etc/resolv.conf | grep nameserver  # What Docker sees
nameserver 127.0.1.1
$ nmcli dev show | grep IP4.DNS  # My actual DNS server
IP4.DNS[1]:                             10.1.1.3

and so Docker containers always default to using 8.8.8.8 rather than using the same DNS server as the host OS.

In my office network external DNS servers are blocked, and so me and my whole team are finding docker containers failing with obscure errors which result from failing to resolve DNS.

There is a workaround, which I help walk all my team members through so they can still use Docker. Although it's frustrating that whenever Docker gets updated, it seems the daemon config file is overwritten and we have to implement the fix all over again. This is less of a problem for me, but is causing significant trouble for our less systems-focused developers.

Is there any hope that Docker might be able to intelligently pick up the network's DNS server in the future?

Steps to reproduce the issue

  1. Use a host OS that makes use of dnsmasq (e.g. Ubuntu since 12.04)
  2. Connect to a network that blocks access to external DNS servers like 8.8.8.8
  3. Try to resolve DNS with Docker (docker run busybox nslookup google.com)

What I get

$ docker run busybox nslookup google.com
Server:    8.8.8.8
Address 1: 8.8.8.8

nslookup: can't resolve 'google.com'

What I expected

$ docker run busybox nslookup google.com
Server:    10.1.1.3
Address 1: 10.1.1.3

Name:      google.com
Address 1: 2a00:1450:4009:811::200e lhr26s02-in-x200e.1e100.net
Address 2: 216.58.198.174 lhr25s10-in-f14.1e100.net

System information

I'm running Xenial natively on a Dell XPS 13 9350.

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:43:49 2016
 OS/Arch:      linux/amd64

$ docker info
Containers: 61
 Running: 3
 Paused: 0
 Stopped: 58
Images: 421
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 441
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-24-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.54 GiB
Name: xps
ID: G4MN:GTXD:4KZP:PTZC:DBYK:WOLA:R3GF:TKLW:ZOOX:NXZT:ALNG:F22D
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: nottrobin
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
@nathanleclaire
Copy link
Contributor

@nathanleclaire nathanleclaire commented Jun 23, 2016

Loading

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jun 24, 2016

The systemd unit file should not be edited directly.
Systemd has a facility called "drop-ins" that allow you to override/augment the unit file.
https://docs.docker.com/engine/admin/systemd/

In addition, it would probably be best to configure the docker daemon via it's config file rather than the systemd unit file. By default docker reads config from /etc/docker/daemon.json. https://docs.docker.com/engine/reference/commandline/dockerd/#linux-configuration-file

Loading

@nottrobin
Copy link
Author

@nottrobin nottrobin commented Jun 24, 2016

Ah! Wonderful suggestions. Thanks @cpuguy83, I'll update my blog post.

Loading

@madpipeline
Copy link

@madpipeline madpipeline commented Jun 30, 2016

I've solved this by adding a daemon configuration file. But I would sure love for this to be implemented.

Loading

@iflowfor8hours
Copy link

@iflowfor8hours iflowfor8hours commented Aug 20, 2016

@cpuguy83 Nice one, I didn't know about the /etc/docker/daemon.json. Might that file get created in a default install to signal to users this is where configuration should live? I went straight for the systemd drop-ins.

Loading

@justincormack
Copy link
Contributor

@justincormack justincormack commented Aug 20, 2016

It is not very easy to implement this as docker has no way to know the
upstream servers. If you set your local DNS relay on localhost to also
listen on the docker0 IP and put that address in resolv.conf that would
also work.

On 30 Jun 2016 3:12 p.m., "Ovidiu-Florin BOGDAN" notifications@github.com
wrote:

I've solved this by adding a daemon configuration file. But I would sure
love for this to be implemented.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#23910 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAdcPFlthUzL2cze4iV_wGe0xRFhQPRpks5qQ87WgaJpZM4I9Lm8
.

Loading

@nottrobin
Copy link
Author

@nottrobin nottrobin commented Sep 9, 2016

@justincormack I can discover the DNS servers by doing nmcli dev show | grep DNS. Couldn't docker do something similar? I mean, more generally, if the system knows its DNS servers (which it must) then surely docker can discover them somehow.

Loading

@awilkins
Copy link

@awilkins awilkins commented Sep 19, 2016

On Ubuntu, I settled on doing this ( as @justincormack describes) by...

  • Adding a /etc/docker/daemon.json that passed in the docker0 bridge IP as a DNS server
  • Adding a /etc/NetworkManager/dnsmasq.d/docker-bridge.conf file that directs the NM-managed DNS cache to listen to the docker0 bridge IP as well as 127.0.1.1

Now container DNS lookups go through your local dnsmasq instance, so DNS in the container should work just as well as DNS on the host.

@nottrobin - my problem specifically was that my system did know it's DNS servers and was passing them to the container - but because those servers were on a VPN, there was no route to them from inside the container, hence all DNS lookups failed. This was partly down to the non-NetworkManager based VPN client we're using that overwites resolv.conf. Turned that "feature" off and instead added config for that subdomain on NetworkManager's dnsmasq itself.

Loading

@justincormack
Copy link
Contributor

@justincormack justincormack commented Sep 20, 2016

@nottrobin nmcli only applies to systems running NetworkManager, there is not an equivalent for other setups.

Loading

@iraadit
Copy link

@iraadit iraadit commented Oct 10, 2016

Thank you A LOT for the workaround @nottrobin, working great even when doing "docker build"
It had been three days of looking for a solution without success before finding your post

Loading

@bsingr
Copy link

@bsingr bsingr commented Nov 23, 2016

Your solution works @awilkins.

Additionally it requires to allow port 53 traffic on the ubuntu firewall for the docker0 bridge ip.

Quick test: disable the ubuntu firewall completely with sudo ufw disable

Loading

@krak3n
Copy link

@krak3n krak3n commented Dec 1, 2016

@bsingr what rule did you use? I've tried loads and can't get it to work on Ubuntu 16.04 - i've resorted to host network mode 😭

Loading

@sanimej
Copy link

@sanimej sanimej commented Dec 19, 2016

@nottrobin Instead of trying to detect the external resolver used by the host resolver the better approach is to allow the containers' DNS queries to be forwarded to the host resolver if the resolv.conf has a loopback address. Using the host resolver has other use cases as well. Its doable with the docker embedded DNS server which is used by default for user defined networks. I pushed a PR in libnetwork for this.

Loading

@butlermd
Copy link

@butlermd butlermd commented Jan 9, 2017

@awilkins or @bsingr, do either of you have more details about how you configured docker-bridge.conf? The daemon.json file seems like it should be simple enough:

{
  "dns": [
    "172.17.0.1"
  ]
}

But I'm unclear on how you configured dnsmasq.

Loading

@awilkins
Copy link

@awilkins awilkins commented Jan 9, 2017

On ubuntu, add a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf

listen-address=172.17.0.1

By default it only listens to DNS requests from 127.0.0.1 (ie, your computer). This tells it to listen to the docker bridge also.

Loading

@butlermd
Copy link

@butlermd butlermd commented Jan 9, 2017

@awilkins Thanks for the clarification!

Loading

@awilkins
Copy link

@awilkins awilkins commented Jan 9, 2017

Also looks like a PR with support for doing this less cryptically landed a few days ago in libnetwork

AFAIK though this won't work for the docker0 network bridge.

moby/libnetwork@1ed42d1

Loading

@sanimej
Copy link

@sanimej sanimej commented Jan 9, 2017

@awilkins Yes, it doesn't work for docker0 bridge because the Docker DNS server is used only for the user created networks. The fix to forward the queries to host loopback resolver relies on the embedded DNS server.

Loading

@dqminh
Copy link
Contributor

@dqminh dqminh commented Jan 16, 2017

@sanimej just to make sure i understand the patch correctly, it only allows docker embedded DNS server to forward queries to the local nameserver ?

We are using https://github.com/gliderlabs/hostlocal, which will still be required if we dont want to use docker embedded DNS server, is that correct ?

Loading

@awilkins
Copy link

@awilkins awilkins commented Jul 25, 2018

As someone else who's run into this : (see my answer on askubuntu...

Ubuntu 18.04 exacerbates this by replacing the NetworkManager managed instance of dnsmasq with systemd-resolved.

systemd-resolved doesn't listen to the docker0 network bridge adapter - and can't be configured to do so, it will only listen on it's hardcoded loopback address.

I worked around this on Ubuntu 18.04 by

  • Disabling the local UDP server on systemd-resolved
  • Installing the NetworkManager dnsmasq support again
  • Reconfiguring it as described in my answer but also
  • Reconfiguring it's primary listen address to 127.0.0.53 (because something in systemd still writes this to /etc/resolvconf and I couldn't work out how to disable that)

Loading

@dashesy
Copy link

@dashesy dashesy commented Jul 25, 2018

@awilkins this sounds like a bug, in systemd. Have you tried opening a bug with them? sure there are more people who use docker and want to try Ubuntu 18 host!

Loading

@awilkins
Copy link

@awilkins awilkins commented Jul 25, 2018

It appears to be a design choice, rather than a bug :

systemd-resolved deliberately only binds to the loopback adapter

This seems to work properly now ... if you don't use the default network. Have just tested this and using a non-default network results in /etc/resolv.conf in the container using 127.0.0.11 - which is docker's embedded DNS resolver. Docker doesn't do this in the default network because of backward-compatibility.

While this is a PITA it's work-around-able, by specifying a non-default network. Mostly I use docker-compose so with user-defined networks defined in my compose files so this will happen by default. docker build and other ad-hoc docker commands accept --net params.

If you want the total convenience of it Just Working™, there's still the option of turning off that systemd-resolved stub and installing dnsmasq (using the NetworkManager one is still easiest for me on 18.04)


The main gotcha with the above - docker-compose runs containers in a user-defined network, but seems to build them in the default network, so docker-compose up will still fail for the above reasons if it needs to build an image that needs DNS requests to build.

Loading

@jerrac
Copy link

@jerrac jerrac commented Jul 25, 2018

@awilkins I did try docker network create ... and then using that custom bridge network when I used docker run. I still ran into the dns issues.

Oddly, if I try the same thing using docker-compose, and a named network, the dns issues go away.

So, what's the different between a docker-compose named network and a docker network create network?

Loading

@awilkins
Copy link

@awilkins awilkins commented Jul 26, 2018

That seems to work fine for me. Is the problem docker's excruciating pickiness about argument order?

$ docker network create cheese
db2e8491afea95700120a3615d01e7507c45cfe2b34397641eb286a1de4ed44c
$ docker run --net cheese bash cat /etc/resolv.conf
nameserver 127.0.0.11
$ docker run  bash cat /etc/resolv.conf
nameserver 172.17.0.1

Loading

@venkat-koppisetti
Copy link

@venkat-koppisetti venkat-koppisetti commented Aug 20, 2018

Docker container should not resolve DNS in Ubuntu 18.04
I want to access server with name from inside container
Am running a container with --dns flag with host's DNS server IP
docker run -itd --dns 127.0.0.53 --name test_img

But from inside container am not able access google server

[root@0b86fa7dde50 scripts]# curl google.com
curl: (6) Could not resolve host: google.com; Unknown error
[root@0b86fa7dde50 scripts]#

Host DNS details:
root@rebaca# nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: google.com
Address: 216.58.194.206
Name: google.com
Address: 2607:f8b0:4005:805::200e

without that --dns flag in docker run command, am bale to access google.com as because by default container is running with 8.8.8.8 dns server

But am not able to access my consul server with Domain name.

How would i access consul server and google server with domain names without adding details in /etc/hosts file

docker info:
root@rebaca:~# docker info
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1012-azure
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.768GiB
Name: azuki-container-test-ehesste
ID: CRS6:TA4X:OTTN:X6TW:GCW5:D5U5:KUTH:JFFB:OF24:FAJ7:WOMF:YIJH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

root@rebaca:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Loading

@alexsuter
Copy link

@alexsuter alexsuter commented Aug 20, 2018

Same problem here. I use docker containers to build my software. They need access to internal build repository, which are not accessible, without an internal name server. How to setup docker properly which uses an internal nameserver?

Loading

@jcberthon
Copy link
Contributor

@jcberthon jcberthon commented Aug 23, 2018

I have read the Docker documentation on DNS and containers. So if the /etc/resolv.conf contains only localhost IPs (like 127.0.0.53 when one uses systemd-resolved which seems default on Ubuntu 18.04 at least), Docker will drop those and not have any localhost IP, so defaulting to 8.8.8.8 (btw not the best solution in terms of privacy...).

However reading the documentation of systemd-resolved one can see that they are maintaining another file /run/systemd/resolve/resolv.conf (at least that's the path on Ubuntu) with all known DNS server. They do not recommend using it directly as it by-passes their service. But what would be reasonable algorithm for Docker would be:

  1. Read /etc/resolv.conf and list all non-localhost IP
  2. If the list of DNS resolver is empty, then check file /run/systemd/resolve/resolv.conf and list all non-localhost IP
  3. If the list is empty use 9.9.9.9 or 1.1.1.1 or 8.8.8.8

That way on Ubuntu (and other systems using systemd-resolved) Docker will still find the right DNS server to use.

Loading

@dashesy
Copy link

@dashesy dashesy commented Aug 23, 2018

@jcberthon if it is a symlink /run/systemd/resolve/resolv.conf it should already work. Why is it not recommended to use it?

Loading

@awilkins
Copy link

@awilkins awilkins commented Aug 24, 2018

@dashesy Because systemd doesn't want you to read /etc/resolv.conf either because it implies you're doing your own DNS lookups. It wants all DNS lookups to use either the the NSS glibc getaddrinfo() call, or use it's dbus API.

I would honestly prefer (especially on my workstation) all DNS lookups from docker containers to be proxied through the host, because it means that if I e.g. sign into a VPN with it's own search domain and DNS service, then docker containers will just be able to resolve names in there without any additional effort.

e.g. the scenario where you sign into a VPN and it splats your /etc/resolv.conf with the DNS server inside the VPN. (this is a badly behaved VPN client that doesn't integrate with NetworkManager).

  • My host can now correctly do lookups of host names inside the VPN
  • Docker containers in the default network will receive the VPN's DNS server in their /etc/resolv.host because it's not a localhost address
  • BUT Because there is no route from the docker0 bridge network to the VPN, all DNS lookups will now timeout inside containers in a default network
  • Containers in a user network will work properly because DNS lookups are proxied through Docker and looked up on the host

My suggested solution to this would be

  • A flag for the docker daemon that switches on the DNS proxy for default networks as well as user-defined ones

My comments on docker's libnetwork reflect this and hopefully someone will implement it (if I had time, I probably would have a stab at it, but my Golang is not strong and I'm busy... ).

Loading

@ImTheDeveloper
Copy link

@ImTheDeveloper ImTheDeveloper commented Mar 5, 2019

I believe I'm seeing the same issues here. Ubuntu 18.04 with a slow dns resolution first time round https://forums.docker.com/t/docker-dns-resolution-slow-for-http-calls-ipv6-bridge-network-issue/70390 I posted my findings here so far.

Loading

@daveneeley
Copy link

@daveneeley daveneeley commented Jul 20, 2019

Seeing this on all docker bridge networks on Centos 7 (7.6.1810), docker 18.09.8

  • do not have dnsmasq
  • do not have systemd-resolved
  • configured /etc/docker/daemon.json with custom nameservers, validated they appear in /etc/resolv.conf within the container
  • host has custom search domain in /etc/resolv.conf, which is passed along to containers
# default bridge network
docker run --rm alpine cat /etc/resolv.conf
search mydomain
nameserver 1.1.1.1
nameserver 8.8.8.8

docker run --rm alpine ping google.com
ping: bad address 'google.com'

#custom bridge network
docker run --net=myexternal --rm alpine cat /etc/resolv.conf
search mydomain
nameserver 127.0.0.11
options ndots:0

docker run --net=myexternal --rm alpine ping google.com
ping: bad address 'google.com'

#host network
docker run --net=host --rm alpine cat /etc/resolv.conf
search mydomain
nameserver 1.1.1.1
nameserver 8.8.8.8

docker run --net=host --rm alpine ping -c 1 google.com
PING google.com (216.58.194.174): 56 data bytes
64 bytes from 216.58.194.174: seq=0 ttl=53 time=36.909 ms

Loading

@rulatir
Copy link

@rulatir rulatir commented May 10, 2021

This bug has delayed our project by a whopping 1.5 years. One release criteria that we have is that the tests must run all green on ALL development machines before we deploy. One development machine is a Ubuntu. Scenario tests that involve making requests to domains resolved by host dnsmasq fail. We have never solved it. We are completely stuck. There is no solution. Some problems simply don't have a solution, and if you happen to have such a problem, then you will fail. Sometimes people fail. Not everyone succeeds. This is life.

Loading

@rulatir
Copy link

@rulatir rulatir commented May 10, 2021

docker run --net=host

So the ONLY solution is to be limited to --net=host and be completely locked out of ever being able to use any other networking modes?

Loading

@moy
Copy link

@moy moy commented May 10, 2021

docker run --net=host

So the ONLY solution is to be limited to --net=host and be completely locked out of ever being able to use any other networking modes?

Not the only solution. I've documented a set of "solutions" (or at least workarounds) here: https://stackoverflow.com/q/49998099/4830165 . I'd like this to be fixed in Docker too, but there are ways to live with the issue.

Loading

@rulatir
Copy link

@rulatir rulatir commented May 10, 2021

None of those work for me.

Loading

@awilkins
Copy link

@awilkins awilkins commented Jun 7, 2021

@rulatir

  1. Disable systemd-resolved
  2. Enable dnsmasq
  3. Configure dnsmasq to listen to the Docker network bridge
  4. Configure Docker to use the network bridge as it's DNS service in default network

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo rm /etc/resolv.conf

Edit /etc/NetworkManager/NetworkManager.conf to add the dns setting to [main]

e.g.

[main]
plugins=ifupdown,keyfile
dns=dnsmasq

Add this file @ /etc/NetworkManager/dnsmasq.d/docker-bridge.conf

listen-address=172.17.0.1

Add this setting to /etc/docker/daemon.json

{
	"dns": [
		"172.17.0.1"
	]
}

Restart stuff.

sudo systemctl restart docker
sudo systemctl restart network-manager

Check you didn't break DNS.

nslookup www.bbc.co.uk

Check you fixed your problem

docker run -it tutum/dnsutils nslookup <my.special.domain>

Loading

@rulatir
Copy link

@rulatir rulatir commented Jun 9, 2021

I ended up doing something similar and it seems to work, but it does require that dnsmasq.service be After=docker.service, otherwise dnsmasq tries to listen on an interface that hasn't been created yet, and fails to start.

Loading

@awilkins
Copy link

@awilkins awilkins commented Jun 10, 2021

I don't have that problem here with the instance of dnsmasq that NetworkManager manages, rather than one managed by systemd, I don't know whether this is down to service start ordering, or whether NetworkManager will restart dnsmasq if it fails.

Loading

@rfay
Copy link

@rfay rfay commented Sep 4, 2021

A warning: If you change the dns setting to something other than the default, host.docker.internal does not resolve.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.