Proposal: DNS service discovery in Docker to replace /etc/hosts #767

Closed
sanimej opened this Issue Nov 21, 2015 · 25 comments

Comments

Projects
None yet
@sanimej
Contributor

sanimej commented Nov 21, 2015

Service discovery in Docker currently relies on updating the container's /etc/hosts file. This resulted in issues like corrupted /etc/hosts file in some cases #17190. Its also not a scalable approach.

This proposal outlines the design for a DNS based service discovery in Docker with the following goals..

  • replace the /etc/hosts baed approach as seamlessly as possible
  • handle the name resolution in docker(libnetwork) without...
    1. duplicating the host/endpoint/IP info elsewhere
    2. requiring changes to network drivers.

Going forward we will support a pluggable DNS model (similar to IPAM). But its not covered in this proposal.

Docker daemon will act as a embedded DNS server for all the containers in a host. From the container point of view nameserver will be available at the loopback address. When the container is launched the loopback address will be the only nameserver populated in /etc/resolv.conf file. DNS queries from the container will be handled by the Docker process.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to. For a qualified name container_name.network_name resolution will be done only if the container is attached to that network.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

The embedded DNS server will be enabled for all user created networks. /etc/hosts file will not be updated for containers on those networks. Only exception is the
containers in the default bridge network (docker0 bridge) with the link option which will continue to use /etc/hosts file. This is to provide backward compatibility for apps relying on that behavior.

The embedded DNS server in Docker is to resolve only the service names offered by Docker containers (local or across hosts). It can't replace any other local or non-local DNS servers used to resolve other services/host names in the cluster.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Nov 23, 2015

Contributor

ping @phemmer

Contributor

mavenugo commented Nov 23, 2015

ping @phemmer

@dave-tucker

This comment has been minimized.

Show comment
Hide comment
@dave-tucker

dave-tucker Nov 23, 2015

Member

👍
Only one question, what if a container has a name of host.example.com, does this hit the embedded DNS server first or is it forwarded to the external servers?

Member

dave-tucker commented Nov 23, 2015

👍
Only one question, what if a container has a name of host.example.com, does this hit the embedded DNS server first or is it forwarded to the external servers?

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Nov 23, 2015

From the container point of view nameserver will be available at the loopback address

I strongly advise against this. By implementing this behavior, you are preventing users from running DNS servers within docker.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to.

I think this should be avoided as well. If you want the container to search multiple networks, adjust the resolv.conf to search those networks.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

 

The 2nd & 3rd items are actually related. In a nutshell, both boil down to the idea that docker provided DNS records should live within their own domain. Meaning the full DNS record for a container would be something like e15fb5654664.docker.local or e15fb5654664.user-network. This way we're sticking with standard DNS design that a single domain does not have multiple authoritative servers with different results.

phemmer commented Nov 23, 2015

From the container point of view nameserver will be available at the loopback address

I strongly advise against this. By implementing this behavior, you are preventing users from running DNS servers within docker.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to.

I think this should be avoided as well. If you want the container to search multiple networks, adjust the resolv.conf to search those networks.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

 

The 2nd & 3rd items are actually related. In a nutshell, both boil down to the idea that docker provided DNS records should live within their own domain. Meaning the full DNS record for a container would be something like e15fb5654664.docker.local or e15fb5654664.user-network. This way we're sticking with standard DNS design that a single domain does not have multiple authoritative servers with different results.

@aidanhs

This comment has been minimized.

Show comment
Hide comment
@aidanhs

aidanhs Nov 23, 2015

Contributor

From the container point of view nameserver will be available at the loopback address.

As @phemmer says, this isn't ok. The same thing was said on the docker-dev list - https://groups.google.com/d/msg/docker-dev/WXkMiPJqh7I/onY0k7yLAwAJ

Contributor

aidanhs commented Nov 23, 2015

From the container point of view nameserver will be available at the loopback address.

As @phemmer says, this isn't ok. The same thing was said on the docker-dev list - https://groups.google.com/d/msg/docker-dev/WXkMiPJqh7I/onY0k7yLAwAJ

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Nov 23, 2015

Contributor

@phemmer @aidanhs Yes, @thockin brought up the same issue. Though the embedded dns server will bind only to loopback:53 it will break cases where a DNS server is doing *:53 bind in the container. We will look at the possibilities to avoid this issue.

Contributor

sanimej commented Nov 23, 2015

@phemmer @aidanhs Yes, @thockin brought up the same issue. Though the embedded dns server will bind only to loopback:53 it will break cases where a DNS server is doing *:53 bind in the container. We will look at the possibilities to avoid this issue.

@mariusGundersen

This comment has been minimized.

Show comment
Hide comment
@mariusGundersen

mariusGundersen Nov 23, 2015

Couple of (maybe stupid) questions I have:

  • what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?
  • will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.
  • how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

Couple of (maybe stupid) questions I have:

  • what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?
  • will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.
  • how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.
@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Nov 23, 2015

Contributor

@mariusGundersen

what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?

For the Address records it can be 1 hour. We can even go lower because this is only for the records in the namespace of docker networks; Hence its going to be a local lookup.

will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

The intent is to seamlessly replace the current /etc/hosts based service discovery. So the user experience will not change. Pluggable DNS will be added after 1.10 release.

Contributor

sanimej commented Nov 23, 2015

@mariusGundersen

what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?

For the Address records it can be 1 hour. We can even go lower because this is only for the records in the namespace of docker networks; Hence its going to be a local lookup.

will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

The intent is to seamlessly replace the current /etc/hosts based service discovery. So the user experience will not change. Pluggable DNS will be added after 1.10 release.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Nov 23, 2015

Contributor

@phemmer

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

Yes, that is what I implied because the docker DNS server can only resolve the names in the domain of docker networks. I will reword it to be explicit.

Contributor

sanimej commented Nov 23, 2015

@phemmer

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

Yes, that is what I implied because the docker DNS server can only resolve the names in the domain of docker networks. I will reword it to be explicit.

@dave-tucker

This comment has been minimized.

Show comment
Hide comment
@dave-tucker

dave-tucker Nov 23, 2015

Member

@sanimej

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

That would be a mistake. We should have never confused container name and hostname. We should fix that behaviour in 1.10:

  • --name is supposed to be a friendly name that can be used instead of the container ID with no significance inside the container
  • --hostname the containers network name

I know why we made the choice we did (--link blurring the lines by using the --name in the link) but it's very wrong. I already met a few early adopters that were hitting collisions at DockerCon.

Member

dave-tucker commented Nov 23, 2015

@sanimej

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

That would be a mistake. We should have never confused container name and hostname. We should fix that behaviour in 1.10:

  • --name is supposed to be a friendly name that can be used instead of the container ID with no significance inside the container
  • --hostname the containers network name

I know why we made the choice we did (--link blurring the lines by using the --name in the link) but it's very wrong. I already met a few early adopters that were hitting collisions at DockerCon.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Nov 23, 2015

Contributor

@dave-tucker @sanimej lets have all the discussions around --name, --link, --hostname or alias in #737. This proposal is to migrate the SD from /etc/hosts to DNS based solution. The actual content that goes inside should be part of #737 discussion.

Contributor

mavenugo commented Nov 23, 2015

@dave-tucker @sanimej lets have all the discussions around --name, --link, --hostname or alias in #737. This proposal is to migrate the SD from /etc/hosts to DNS based solution. The actual content that goes inside should be part of #737 discussion.

@fermayo

This comment has been minimized.

Show comment
Hide comment
@fermayo

fermayo Nov 27, 2015

+1 to replace updating /etc/hosts with an embedded DNS server.

Does the engine have an IP on all networks that it creates? Couldn't we just use that IP as the nameserver set on the /etc/resolv.conf?
Regarding the scoping of the resolution: if each network is a DNS domain, can't we just have all container's network names as search domains so short hostname resolution works out of the box? Or at least provide a default domain for the containers so they don't need to know any network names.

fermayo commented Nov 27, 2015

+1 to replace updating /etc/hosts with an embedded DNS server.

Does the engine have an IP on all networks that it creates? Couldn't we just use that IP as the nameserver set on the /etc/resolv.conf?
Regarding the scoping of the resolution: if each network is a DNS domain, can't we just have all container's network names as search domains so short hostname resolution works out of the box? Or at least provide a default domain for the containers so they don't need to know any network names.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Nov 30, 2015

Contributor

@fermayo No, the engine doesn't have an IP on all the networks. Allocating one IP per host for multi-host networks may not be ideal. If we have to go down this path I think it can be made to work with one IP allocated per network and the using the same IP on all the hosts for DNS resolution; the embedded server is always local.

For the resolution yes, the search keyword can be used. But this will involve additional exchanges between the resolver and the docker DNS server.

Contributor

sanimej commented Nov 30, 2015

@fermayo No, the engine doesn't have an IP on all the networks. Allocating one IP per host for multi-host networks may not be ideal. If we have to go down this path I think it can be made to work with one IP allocated per network and the using the same IP on all the hosts for DNS resolution; the embedded server is always local.

For the resolution yes, the search keyword can be used. But this will involve additional exchanges between the resolver and the docker DNS server.

@smil2k

This comment has been minimized.

Show comment
Hide comment
@smil2k

smil2k Dec 28, 2015

Hi,

Did you consider using the NSS layer of LibC for name resolution? (/etc/nsswitch.conf)

It seems to me that running and routing to a DNS server internally and updating / configuring it seems to be a bigger job than creating a small NSS backend which connects to docker-daemon and asks for the service ip. The communication could be done using a unix socket which is "volumed" in by default somewhere under /var/lib/docker and friends.

Cheers,

Tamas

smil2k commented Dec 28, 2015

Hi,

Did you consider using the NSS layer of LibC for name resolution? (/etc/nsswitch.conf)

It seems to me that running and routing to a DNS server internally and updating / configuring it seems to be a bigger job than creating a small NSS backend which connects to docker-daemon and asks for the service ip. The communication could be done using a unix socket which is "volumed" in by default somewhere under /var/lib/docker and friends.

Cheers,

Tamas

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Dec 28, 2015

@smil2k Relying on nss wouldn't work for a lot of software which doesn't use the libc resolver.

I would also argue that standing up an NSS library would be no easier than an actual DNS server, and I think likely harder.
Standing up a DNS listener is not that hard. The majority of the difficulty in implementing this is going to be the record namespacing logic, which exists whether the request came from NSS or DNS.

phemmer commented Dec 28, 2015

@smil2k Relying on nss wouldn't work for a lot of software which doesn't use the libc resolver.

I would also argue that standing up an NSS library would be no easier than an actual DNS server, and I think likely harder.
Standing up a DNS listener is not that hard. The majority of the difficulty in implementing this is going to be the record namespacing logic, which exists whether the request came from NSS or DNS.

@smil2k

This comment has been minimized.

Show comment
Hide comment
@smil2k

smil2k Dec 28, 2015

@phemmer: Those software which does not use NSS, would not work by adding resolv.conf entries, and never worked with the existing /etc/hosts solution. (as /etc/hosts being red by the "files" NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts (which means stability and less maint effort) than a DNS server, which must be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch it to the existing pile.

smil2k commented Dec 28, 2015

@phemmer: Those software which does not use NSS, would not work by adding resolv.conf entries, and never worked with the existing /etc/hosts solution. (as /etc/hosts being red by the "files" NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts (which means stability and less maint effort) than a DNS server, which must be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch it to the existing pile.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Dec 28, 2015

Contributor

@phemmer @smil2k thanks to @sanimej PR (#841) is open to address this proposal. PTAL

Contributor

mavenugo commented Dec 28, 2015

@phemmer @smil2k thanks to @sanimej PR (#841) is open to address this proposal. PTAL

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Dec 28, 2015

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

phemmer commented Dec 28, 2015

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Dec 28, 2015

@mavenugo thanks for the ref to #841. Took a quick look at it and I see numerous issues :-(
Will do a deeper review on the PR later.

phemmer commented Dec 28, 2015

@mavenugo thanks for the ref to #841. Took a quick look at it and I see numerous issues :-(
Will do a deeper review on the PR later.

@smil2k

This comment has been minimized.

Show comment
Hide comment
@smil2k

smil2k Dec 28, 2015

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

Can you name some examples?
I'm curious which use cases needs to replicate glibc mechanisms.

I could imagine uclibc as one. I really curious, so please don't take is as an offense.

smil2k commented Dec 28, 2015

@smil2k resolv.conf and /etc/hosts can be (and are) read by things other than nss/libc.

Can you name some examples?
I'm curious which use cases needs to replicate glibc mechanisms.

I could imagine uclibc as one. I really curious, so please don't take is as an offense.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Dec 28, 2015

Contributor

@phemmer it would be good if you can share your comments ASAP. we are running short of time for getting this in for the release.

Contributor

mavenugo commented Dec 28, 2015

@phemmer it would be good if you can share your comments ASAP. we are running short of time for getting this in for the release.

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Dec 28, 2015

Contributor

resolv.conf and /etc/hosts way pre-date libnss. Those are pretty universal
steps in UNIX-style name resolution since approximately forever. Libnss is
an abstraction above those to allow other modes of resolution. You can
count on DNS working for almost everyone.

On Mon, Dec 28, 2015 at 8:07 AM, Tamas notifications@github.com wrote:

@phemmer https://github.com/phemmer: Those software which does not use
NSS, would not work by adding resolv.conf entries, and never worked with
the existing /etc/hosts solution. (as /etc/hosts being red by the "files"
NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts
(which means stability and less maint effort) than a DNS server, which must
be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch
it to the existing pile.


Reply to this email directly or view it on GitHub
#767 (comment).

Contributor

thockin commented Dec 28, 2015

resolv.conf and /etc/hosts way pre-date libnss. Those are pretty universal
steps in UNIX-style name resolution since approximately forever. Libnss is
an abstraction above those to allow other modes of resolution. You can
count on DNS working for almost everyone.

On Mon, Dec 28, 2015 at 8:07 AM, Tamas notifications@github.com wrote:

@phemmer https://github.com/phemmer: Those software which does not use
NSS, would not work by adding resolv.conf entries, and never worked with
the existing /etc/hosts solution. (as /etc/hosts being red by the "files"
NSS backend).
Am I missing something here?

An nss plugin is much simpler tool with inherently much less moving parts
(which means stability and less maint effort) than a DNS server, which must
be protected, updated internally, packets must be routed to it etc...

Of course it is EASIER to use an existing go dns server library and stitch
it to the existing pile.


Reply to this email directly or view it on GitHub
#767 (comment).

@clinta

This comment has been minimized.

Show comment
Hide comment
@clinta

clinta Feb 10, 2016

Contributor

Is there a design proposal for what a pluggable system should look like? Would it be appropriate to open a new issue to discuss it? I have some specific needs that I think could be implemented as a plugin, and I don't mind getting started on trying to add some plugin support, but I don't want to invest the time if my ideas are significantly different than the docker team's plans and would not be merged.

Contributor

clinta commented Feb 10, 2016

Is there a design proposal for what a pluggable system should look like? Would it be appropriate to open a new issue to discuss it? I have some specific needs that I think could be implemented as a plugin, and I don't mind getting started on trying to add some plugin support, but I don't want to invest the time if my ideas are significantly different than the docker team's plans and would not be merged.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Feb 10, 2016

Contributor

@clinta can you please open a proposal with your ideas and we can discuss it before you start to implement the plugin.

Contributor

mavenugo commented Feb 10, 2016

@clinta can you please open a proposal with your ideas and we can discuss it before you start to implement the plugin.

@BediarSofiane

This comment has been minimized.

Show comment
Hide comment
@BediarSofiane

BediarSofiane Mar 11, 2016

Hi,
I noticed that this proposal has been officially integrated in v1.10.
I would like to know the details of how the embedded DNS server is implemented (for exemple where containers names are stored, etc). Is there is a doc that I can use to study this more in details?
Thank you guys.

Hi,
I noticed that this proposal has been officially integrated in v1.10.
I would like to know the details of how the embedded DNS server is implemented (for exemple where containers names are stored, etc). Is there is a doc that I can use to study this more in details?
Thank you guys.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
Contributor

mavenugo commented Mar 11, 2016

@vectree vectree referenced this issue in vectree/task.4.lorierp.test.1518366382850 Feb 11, 2018

Open

04 Многоконтейнерные окружения.md #5

@vectree vectree referenced this issue in vectree/task.4.Kuzmrom7.test.1522322155268 Mar 29, 2018

Open

04 Многоконтейнерные окружения.md #5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment