Proposal: DNS service discovery in Docker to replace /etc/hosts

[sanimej]

Service discovery in Docker currently relies on updating the container's /etc/hosts file. This resulted in issues like corrupted /etc/hosts file in some cases #17190. Its also not a scalable approach.

This proposal outlines the design for a DNS based service discovery in Docker with the following goals..

• replace the /etc/hosts baed approach as seamlessly as possible
• handle the name resolution in docker(libnetwork) without...
  1. duplicating the host/endpoint/IP info elsewhere
  2. requiring changes to network drivers.

Going forward we will support a pluggable DNS model (similar to IPAM). But its not covered in this proposal.

Docker daemon will act as a embedded DNS server for all the containers in a host. From the container point of view nameserver will be available at the loopback address. When the container is launched the loopback address will be the only nameserver populated in /etc/resolv.conf file. DNS queries from the container will be handled by the Docker process.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to. For a qualified name container_name.network_name resolution will be done only if the container is attached to that network.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

The embedded DNS server will be enabled for all user created networks. /etc/hosts file will not be updated for containers on those networks. Only exception is the
containers in the default bridge network (docker0 bridge) with the link option which will continue to use /etc/hosts file. This is to provide backward compatibility for apps relying on that behavior.

The embedded DNS server in Docker is to resolve only the service names offered by Docker containers (local or across hosts). It can't replace any other local or non-local DNS servers used to resolve other services/host names in the cluster.

[mavenugo]

ping @phemmer

[dave-tucker]

👍
Only one question, what if a container has a name of host.example.com, does this hit the embedded DNS server first or is it forwarded to the external servers?

[phemmer]

From the container point of view nameserver will be available at the loopback address

I strongly advise against this. By implementing this behavior, you are preventing users from running DNS servers within docker.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to.

I think this should be avoided as well. If you want the container to search multiple networks, adjust the resolv.conf to search those networks.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

 

The 2nd & 3rd items are actually related. In a nutshell, both boil down to the idea that docker provided DNS records should live within their own domain. Meaning the full DNS record for a container would be something like e15fb5654664.docker.local or e15fb5654664.user-network. This way we're sticking with standard DNS design that a single domain does not have multiple authoritative servers with different results.

[aidanhs]

From the container point of view nameserver will be available at the loopback address.

As @phemmer says, this isn't ok. The same thing was said on the docker-dev list - https://groups.google.com/d/msg/docker-dev/WXkMiPJqh7I/onY0k7yLAwAJ

[sanimej]

@phemmer @aidanhs Yes, @thockin brought up the same issue. Though the embedded dns server will bind only to loopback:53 it will break cases where a DNS server is doing *:53 bind in the container. We will look at the possibilities to avoid this issue.

[mariusGundersen]

Couple of (maybe stupid) questions I have:

• what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?
• will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.
• how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

[sanimej]

@mariusGundersen

what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?

For the Address records it can be 1 hour. We can even go lower because this is only for the records in the namespace of docker networks; Hence its going to be a local lookup.

will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

The intent is to seamlessly replace the current /etc/hosts based service discovery. So the user experience will not change. Pluggable DNS will be added after 1.10 release.

[sanimej]

@phemmer

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

Yes, that is what I implied because the docker DNS server can only resolve the names in the domain of docker networks. I will reword it to be explicit.

[dave-tucker]

@sanimej

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

That would be a mistake. We should have never confused container name and hostname. We should fix that behaviour in 1.10:

• --name is supposed to be a friendly name that can be used instead of the container ID with no significance inside the container
• --hostname the containers network name

I know why we made the choice we did (--link blurring the lines by using the --name in the link) but it's very wrong. I already met a few early adopters that were hitting collisions at DockerCon.

[mavenugo]

@dave-tucker @sanimej lets have all the discussions around --name, --link, --hostname or alias in #737. This proposal is to migrate the SD from /etc/hosts to DNS based solution. The actual content that goes inside should be part of #737 discussion.

[fermayo]

+1 to replace updating /etc/hosts with an embedded DNS server.

Does the engine have an IP on all networks that it creates? Couldn't we just use that IP as the nameserver set on the /etc/resolv.conf?
Regarding the scoping of the resolution: if each network is a DNS domain, can't we just have all container's network names as search domains so short hostname resolution works out of the box? Or at least provide a default domain for the containers so they don't need to know any network names.

[sanimej]

@fermayo No, the engine doesn't have an IP on all the networks. Allocating one IP per host for multi-host networks may not be ideal. If we have to go down this path I think it can be made to work with one IP allocated per network and the using the same IP on all the hosts for DNS resolution; the embedded server is always local.

For the resolution yes, the search keyword can be used. But this will involve additional exchanges between the resolver and the docker DNS server.

[smil2k]

Hi,

Did you consider using the NSS layer of LibC for name resolution? (/etc/nsswitch.conf)

It seems to me that running and routing to a DNS server internally and updating / configuring it seems to be a bigger job than creating a small NSS backend which connects to docker-daemon and asks for the service ip. The communication could be done using a unix socket which is "volumed" in by default somewhere under /var/lib/docker and friends.

Cheers,

Tamas

[phemmer]

@smil2k Relying on nss wouldn't work for a lot of software which doesn't use the libc resolver.

I would also argue that standing up an NSS library would be no easier than an actual DNS server, and I think likely harder.
Standing up a DNS listener is not that hard. The majority of the difficulty in implementing this is going to be the record namespacing logic, which exists whether the request came from NSS or DNS.

[smil2k]

@phemmer: Those software which does not use NSS, would not work by adding resolv.conf entries, and never worked with the existing /etc/hosts solution. (as /etc/hosts being red by the "files" NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts (which means stability and less maint effort) than a DNS server, which must be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch it to the existing pile.

[mavenugo]

@phemmer @smil2k thanks to @sanimej PR (#841) is open to address this proposal. PTAL

[phemmer]

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

[phemmer]

@mavenugo thanks for the ref to #841. Took a quick look at it and I see numerous issues :-(
Will do a deeper review on the PR later.

[smil2k]

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

Can you name some examples?
I'm curious which use cases needs to replicate glibc mechanisms.

I could imagine uclibc as one. I really curious, so please don't take is as an offense.

[mavenugo]

@phemmer it would be good if you can share your comments ASAP. we are running short of time for getting this in for the release.

[thockin]

resolv.conf and /etc/hosts way pre-date libnss. Those are pretty universal
steps in UNIX-style name resolution since approximately forever. Libnss is
an abstraction above those to allow other modes of resolution. You can
count on DNS working for almost everyone.

On Mon, Dec 28, 2015 at 8:07 AM, Tamas notifications@github.com wrote:

@phemmer https://github.com/phemmer: Those software which does not use
NSS, would not work by adding resolv.conf entries, and never worked with
the existing /etc/hosts solution. (as /etc/hosts being red by the "files"
NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts
(which means stability and less maint effort) than a DNS server, which must
be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch
it to the existing pile.

—
Reply to this email directly or view it on GitHub
#767 (comment).

[clinta]

Is there a design proposal for what a pluggable system should look like? Would it be appropriate to open a new issue to discuss it? I have some specific needs that I think could be implemented as a plugin, and I don't mind getting started on trying to add some plugin support, but I don't want to invest the time if my ideas are significantly different than the docker team's plans and would not be merged.

[mavenugo]

@clinta can you please open a proposal with your ideas and we can discuss it before you start to implement the plugin.

[BediarSofiane]

Hi,
I noticed that this proposal has been officially integrated in v1.10.
I would like to know the details of how the embedded DNS server is implemented (for exemple where containers names are stored, etc). Is there is a doc that I can use to study this more in details?
Thank you guys.

[mavenugo]

@BediarSofiane you can read more about it in docker docs. https://docs.docker.com/engine/userguide/networking/configure-dns/