New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.10] How to disable embedded DNS? #19474

Closed
ibuildthecloud opened this Issue Jan 20, 2016 · 86 comments

Comments

Projects
None yet
@ibuildthecloud
Contributor

ibuildthecloud commented Jan 20, 2016

Ask you could probably guess, not everybody wants their DNS requests intercepted. How does one disable the embedded DNS server? I'm sure it's in the docs somewhere but I didn't see it.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jan 20, 2016

Contributor

@ibuildthecloud we replaced the /etc/hosts based SD (in 1.9) with embedded DNS. We couldn't make it pluggable in this release. Contributions welcome.

Contributor

mavenugo commented Jan 20, 2016

@ibuildthecloud we replaced the /etc/hosts based SD (in 1.9) with embedded DNS. We couldn't make it pluggable in this release. Contributions welcome.

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

Wait, so any container that wants to use port 53 will fail? Is that what I
am reading?

On Tue, Jan 19, 2016 at 8:47 PM, Madhu Venugopal notifications@github.com
wrote:

@ibuildthecloud https://github.com/ibuildthecloud we replaced the
/etc/hosts based SD (in 1.9) with embedded DNS. We couldn't make it
pluggable in this release. Contributions welcome.


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

Wait, so any container that wants to use port 53 will fail? Is that what I
am reading?

On Tue, Jan 19, 2016 at 8:47 PM, Madhu Venugopal notifications@github.com
wrote:

@ibuildthecloud https://github.com/ibuildthecloud we replaced the
/etc/hosts based SD (in 1.9) with embedded DNS. We couldn't make it
pluggable in this release. Contributions welcome.


Reply to this email directly or view it on GitHub
#19474 (comment).

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jan 20, 2016

Contributor

@thockin I think you should give it a try.

Wait, so any container that wants to use port 53 will fail? Is that what I am reading ?

No. You are reading it incorrectly.

Contributor

mavenugo commented Jan 20, 2016

@thockin I think you should give it a try.

Wait, so any container that wants to use port 53 will fail? Is that what I am reading ?

No. You are reading it incorrectly.

@brendandburns

This comment has been minimized.

Show comment
Hide comment
@brendandburns

brendandburns Jan 20, 2016

@mavenugo so how do I disable the DNS server?

brendandburns commented Jan 20, 2016

@mavenugo so how do I disable the DNS server?

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jan 20, 2016

Contributor

@brendandburns DNS server is disabled by default for all the non-user-defined networks. The embedded DNS server is enabled only for user-defined networks. and it can be expanded with --dns for any external DNS servers.

Can you please explain your use-case ? Is it for k8s ? In that case, I guess you are interested in non-user-defined networks ?.

Contributor

mavenugo commented Jan 20, 2016

@brendandburns DNS server is disabled by default for all the non-user-defined networks. The embedded DNS server is enabled only for user-defined networks. and it can be expanded with --dns for any external DNS servers.

Can you please explain your use-case ? Is it for k8s ? In that case, I guess you are interested in non-user-defined networks ?.

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

We need to consider both "default" network (bridge) and user-defined
networks (bridge, overlay, and 3rd party). Any incursion on ports inside
the containers seems unacceptable.

On Tue, Jan 19, 2016 at 9:14 PM, Madhu Venugopal notifications@github.com
wrote:

@brendandburns https://github.com/brendandburns DNS server is disabled
by default for all the user-defined networks. The embedded DNS server is
enabled only for user-defined networks. and it can be expanded with --dns
for any external DNS servers.

Can you please explain your use-case ? Is it for k8s ? In that case, I
guess you are interested in non-user-defined networks ?.


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

We need to consider both "default" network (bridge) and user-defined
networks (bridge, overlay, and 3rd party). Any incursion on ports inside
the containers seems unacceptable.

On Tue, Jan 19, 2016 at 9:14 PM, Madhu Venugopal notifications@github.com
wrote:

@brendandburns https://github.com/brendandburns DNS server is disabled
by default for all the user-defined networks. The embedded DNS server is
enabled only for user-defined networks. and it can be expanded with --dns
for any external DNS servers.

Can you please explain your use-case ? Is it for k8s ? In that case, I
guess you are interested in non-user-defined networks ?.


Reply to this email directly or view it on GitHub
#19474 (comment).

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

@thockin There is no incursion of port 53 inside the container i.e you can run an actual DNS server inside a container if you want regardless

Contributor

mrjana commented Jan 20, 2016

@thockin There is no incursion of port 53 inside the container i.e you can run an actual DNS server inside a container if you want regardless

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

Hmm, then I mis-parsed how the embedded DNS server works - how does one
access it?

https://github.com/docker/libnetwork/blob/master/resolver.go#L14

On Tue, Jan 19, 2016 at 9:26 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

@thockin https://github.com/thockin There is no incursion of port 53
inside the container i.e you can run an actual DNS server inside a
container if you want regardless


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

Hmm, then I mis-parsed how the embedded DNS server works - how does one
access it?

https://github.com/docker/libnetwork/blob/master/resolver.go#L14

On Tue, Jan 19, 2016 at 9:26 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

@thockin https://github.com/thockin There is no incursion of port 53
inside the container i.e you can run an actual DNS server inside a
container if you want regardless


Reply to this email directly or view it on GitHub
#19474 (comment).

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@thockin /etc/resolv.conf is set as follows

nameserver 127.0.0.11
options ndots:0

And iptables is set to

-A OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:49669
-A POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 49669 -j SNAT --to-source :53

And somehow something magically listens in the network namespace of the container.

Contributor

ibuildthecloud commented Jan 20, 2016

@thockin /etc/resolv.conf is set as follows

nameserver 127.0.0.11
options ndots:0

And iptables is set to

-A OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:49669
-A POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 49669 -j SNAT --to-source :53

And somehow something magically listens in the network namespace of the container.

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@mavenugo Can't we just add a simple flag on network create? Can custom network drivers opt out of the DNS server?

Contributor

ibuildthecloud commented Jan 20, 2016

@mavenugo Can't we just add a simple flag on network create? Can custom network drivers opt out of the DNS server?

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

@ibuildthecloud This just replaces the existing /etc/hosts behavior. There is no functionality change at all. Earlier an /etc/hosts file was intercepting the name resolution requests. Now an embedded DNS server is intercepting the request. What was working before in 1.9 for you that is not going to work now?

Contributor

mrjana commented Jan 20, 2016

@ibuildthecloud This just replaces the existing /etc/hosts behavior. There is no functionality change at all. Earlier an /etc/hosts file was intercepting the name resolution requests. Now an embedded DNS server is intercepting the request. What was working before in 1.9 for you that is not going to work now?

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@mrjana The difference just being that you are putting a component in the critical data path. Editing the host files just uses existing resolver code that has been around for awhile. Putting a DNS server in the path is a whole different ball game. At Rancher we've written our DNS server, (just like everyone has these days. I think it's a new hello world in go), and have already dealt with issues with things we never thought of (compression, odd AAAA records, Windows wackiness). So the point being, you're asking a lot of users to immediately and always trust that your DNS server works, with no fallback.

@mrjana What actually listens for DNS? Is this the docker daemon itself? I'd like to understand how you do the magic of binding to the sock in the networking namespace of the container.

Contributor

ibuildthecloud commented Jan 20, 2016

@mrjana The difference just being that you are putting a component in the critical data path. Editing the host files just uses existing resolver code that has been around for awhile. Putting a DNS server in the path is a whole different ball game. At Rancher we've written our DNS server, (just like everyone has these days. I think it's a new hello world in go), and have already dealt with issues with things we never thought of (compression, odd AAAA records, Windows wackiness). So the point being, you're asking a lot of users to immediately and always trust that your DNS server works, with no fallback.

@mrjana What actually listens for DNS? Is this the docker daemon itself? I'd like to understand how you do the magic of binding to the sock in the networking namespace of the container.

@brendandburns

This comment has been minimized.

Show comment
Hide comment
@brendandburns

brendandburns Jan 20, 2016

To be honest, we really wanted to disable the /etc/hosts behavior too, since it led to corrupt /etc/hosts files (#17190) but that didn't happen.

I agree @ibuildthecloud this is about 1000x more complicated than changing the /etc/hosts file to add some new static hosts, and even that led to corruption. If your DNS server breaks, containers basically stop working entirely, that's a pretty severe risk.

For production users, this has to be disableable.

brendandburns commented Jan 20, 2016

To be honest, we really wanted to disable the /etc/hosts behavior too, since it led to corrupt /etc/hosts files (#17190) but that didn't happen.

I agree @ibuildthecloud this is about 1000x more complicated than changing the /etc/hosts file to add some new static hosts, and even that led to corruption. If your DNS server breaks, containers basically stop working entirely, that's a pretty severe risk.

For production users, this has to be disableable.

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

Hmm, I misparsed the .11 as .1, I guess. Sorry for that. This is ... less
bad in an obvious way, but still has a smell to it. Consuming an ephemeral
port is far less likely to be a problem, at least.

I don't have an RC installed yet - if I pass my own --dns and --dns-search,
does this DNS server get removed or is it prepended or appended to my own
list? Given this bug, I assume it is not replaced. Linux has a hard limit
of 3 nameserver lines, so this is still a significant incursion.

Also - does this embedded server forward to my own DNS servers? We learned
the hard way that nameserver lines should be interchangeable (i.e. not
have 1 that just serves X while another serves Y) or else suffer the
vagaries of the myriad DNS resolvers (and total lack of a spec).

On Tue, Jan 19, 2016 at 9:30 PM, Darren Shepherd notifications@github.com
wrote:

@thockin https://github.com/thockin /etc/resolv.conf is set as follows

nameserver 127.0.0.11
options ndots:0

And iptables is set to

-A OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:49669
-A POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 49669 -j SNAT --to-source :53

And somehow something magically listens in the network namespace of the
container.


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

Hmm, I misparsed the .11 as .1, I guess. Sorry for that. This is ... less
bad in an obvious way, but still has a smell to it. Consuming an ephemeral
port is far less likely to be a problem, at least.

I don't have an RC installed yet - if I pass my own --dns and --dns-search,
does this DNS server get removed or is it prepended or appended to my own
list? Given this bug, I assume it is not replaced. Linux has a hard limit
of 3 nameserver lines, so this is still a significant incursion.

Also - does this embedded server forward to my own DNS servers? We learned
the hard way that nameserver lines should be interchangeable (i.e. not
have 1 that just serves X while another serves Y) or else suffer the
vagaries of the myriad DNS resolvers (and total lack of a spec).

On Tue, Jan 19, 2016 at 9:30 PM, Darren Shepherd notifications@github.com
wrote:

@thockin https://github.com/thockin /etc/resolv.conf is set as follows

nameserver 127.0.0.11
options ndots:0

And iptables is set to

-A OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:49669
-A POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 49669 -j SNAT --to-source :53

And somehow something magically listens in the network namespace of the
container.


Reply to this email directly or view it on GitHub
#19474 (comment).

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@thockin @brendandburns For the default network, the "bridge" network, the embedded DNS server is not enabled. This is only if you do docker network create foo, then foo will have the embedded DNS server enabled. I don't think there is any immediate impact on k8s, just that this stands as another barrier to k8s using Docker network drivers.

Contributor

ibuildthecloud commented Jan 20, 2016

@thockin @brendandburns For the default network, the "bridge" network, the embedded DNS server is not enabled. This is only if you do docker network create foo, then foo will have the embedded DNS server enabled. I don't think there is any immediate impact on k8s, just that this stands as another barrier to k8s using Docker network drivers.

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

The embedded DNS server is really a light weight server only interested in responding to container name/alias queries (which were earlier present as /etc/hosts records) and for everything else it just acts as the proxy to any other DNS server you have configured including Rancher's own DNS server if that's what you want your users to use.

We create a socket in the container namespace and listen on it in the daemon itself and respond queries using that. The socket acts as the demux identifier to identify the network in case of unqualified name queries.

Contributor

mrjana commented Jan 20, 2016

The embedded DNS server is really a light weight server only interested in responding to container name/alias queries (which were earlier present as /etc/hosts records) and for everything else it just acts as the proxy to any other DNS server you have configured including Rancher's own DNS server if that's what you want your users to use.

We create a socket in the container namespace and listen on it in the daemon itself and respond queries using that. The socket acts as the demux identifier to identify the network in case of unqualified name queries.

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

@mrjana

a) how do I configure what server(s) it forwards to?
b) how do I configure upstream timeouts?
c) I still don't want it - it's antithetical to our setup, and we really
just want to bypass it

On Tue, Jan 19, 2016 at 9:47 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

The embedded DNS server is really a light weight server only interested in
responding to container name/alias queries (which were earlier present as
/etc/hosts records) and for everything else it just acts as the proxy to
any other DNS server you have configured including Rancher's own DNS server
if that's what you want your users to use.

We create a socket in the container namespace and listen on it in the
daemon itself and respond queries using that. The socket acts as the demux
identifier to identify the network in case of unqualified name queries.


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

@mrjana

a) how do I configure what server(s) it forwards to?
b) how do I configure upstream timeouts?
c) I still don't want it - it's antithetical to our setup, and we really
just want to bypass it

On Tue, Jan 19, 2016 at 9:47 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

The embedded DNS server is really a light weight server only interested in
responding to container name/alias queries (which were earlier present as
/etc/hosts records) and for everything else it just acts as the proxy to
any other DNS server you have configured including Rancher's own DNS server
if that's what you want your users to use.

We create a socket in the container namespace and listen on it in the
daemon itself and respond queries using that. The socket acts as the demux
identifier to identify the network in case of unqualified name queries.


Reply to this email directly or view it on GitHub
#19474 (comment).

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@mrjana Yeah, I know, it's based on the same library everyone uses https://github.com/miekg/dns. Still a shame you can't disable it. If you don't need it, why put the daemon in the critical path.

Contributor

ibuildthecloud commented Jan 20, 2016

@mrjana Yeah, I know, it's based on the same library everyone uses https://github.com/miekg/dns. Still a shame you can't disable it. If you don't need it, why put the daemon in the critical path.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Jan 20, 2016

@thockin

Also - does this embedded server forward to my own DNS servers?

Yes. If you pass your servers embedded server will forward the queries to the configured servers if the name can't be resolved in the docker network domain.

We learned the hard way that nameserver lines should be interchangeable (i.e. not have 1 that just serves X while another serves Y) or else suffer the vagaries of the myriad DNS resolvers (and total lack of a spec).

Yes, you can't have different nameservers in resolv.conf serving different subset of the domains. resolv.conf file will have only the embedded server listed.

Linux has a hard limit of 3 nameserver lines, so this is still a significant incursion.

There is no incursion. Embedded DNS server will allow up to 3 nameservers. PR for this is in the works.

sanimej commented Jan 20, 2016

@thockin

Also - does this embedded server forward to my own DNS servers?

Yes. If you pass your servers embedded server will forward the queries to the configured servers if the name can't be resolved in the docker network domain.

We learned the hard way that nameserver lines should be interchangeable (i.e. not have 1 that just serves X while another serves Y) or else suffer the vagaries of the myriad DNS resolvers (and total lack of a spec).

Yes, you can't have different nameservers in resolv.conf serving different subset of the domains. resolv.conf file will have only the embedded server listed.

Linux has a hard limit of 3 nameserver lines, so this is still a significant incursion.

There is no incursion. Embedded DNS server will allow up to 3 nameservers. PR for this is in the works.

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

@thockin Whatever you do to configure DNS options in docker will work as is.

Contributor

mrjana commented Jan 20, 2016

@thockin Whatever you do to configure DNS options in docker will work as is.

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

@mrjana I see the endpoint has a flag, disableResolution. Can a network driver set this to true?

Contributor

ibuildthecloud commented Jan 20, 2016

@mrjana I see the endpoint has a flag, disableResolution. Can a network driver set this to true?

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Jan 20, 2016

Contributor

...but there is not and will not be a way to opt out of this functionality?

On Tue, Jan 19, 2016 at 9:56 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

@thockin https://github.com/thockin Whatever you do to configure DNS
options in docker will work as is.


Reply to this email directly or view it on GitHub
#19474 (comment).

Contributor

thockin commented Jan 20, 2016

...but there is not and will not be a way to opt out of this functionality?

On Tue, Jan 19, 2016 at 9:56 PM, Jana Radhakrishnan <
notifications@github.com> wrote:

@thockin https://github.com/thockin Whatever you do to configure DNS
options in docker will work as is.


Reply to this email directly or view it on GitHub
#19474 (comment).

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

@ibuildthecloud Disabling name discovery by docker on a per-network basis is a different question and one which has been brought up even during 1.9 time frame when we were doing it using /etc/hosts. The decision on that still remains the same. If you don't want name discovery use default bridge network. Else you chose to use the name discovery.

Contributor

mrjana commented Jan 20, 2016

@ibuildthecloud Disabling name discovery by docker on a per-network basis is a different question and one which has been brought up even during 1.9 time frame when we were doing it using /etc/hosts. The decision on that still remains the same. If you don't want name discovery use default bridge network. Else you chose to use the name discovery.

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Jan 20, 2016

Contributor

@ibuildthecloud that flag is not exposed as a public api

Contributor

mrjana commented Jan 20, 2016

@ibuildthecloud that flag is not exposed as a public api

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

If you want name discovery use default bridge network. Else you chose to use the name discovery.

Harsh, but okay, you make the rules.

Contributor

ibuildthecloud commented Jan 20, 2016

If you want name discovery use default bridge network. Else you chose to use the name discovery.

Harsh, but okay, you make the rules.

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jan 20, 2016

Contributor

@ibuildthecloud @thockin we all understand that service discovery is a fundamental feature and if I parse your questions correctly, it is not about disabling the feature, but it is about making it pluggable. As per any docker plugin philosophy, we are approaching it as batteries included by swappable. But we couldn't get to implement the "swappable" part in 1.10 as we prioritized replacing /etc/hosts with DNS as pointed by all of you in the 1.9 release. This effort has been underway in libnetwork for more than few weeks, but we didn't receive any contributions regarding the pluggability for SD.

As seen in the past, providing a way to disable a functionality, just encourages more wrappers and that results in fragmentation & impacts the user experience badly. But, having a pluggable implementation gives the correct balance of functionality and user experience. We are seeing that in action in Volumes and network plugins already.

am highly in favor of a pluggable SD approach and any contributions to that is welcome.

Contributor

mavenugo commented Jan 20, 2016

@ibuildthecloud @thockin we all understand that service discovery is a fundamental feature and if I parse your questions correctly, it is not about disabling the feature, but it is about making it pluggable. As per any docker plugin philosophy, we are approaching it as batteries included by swappable. But we couldn't get to implement the "swappable" part in 1.10 as we prioritized replacing /etc/hosts with DNS as pointed by all of you in the 1.9 release. This effort has been underway in libnetwork for more than few weeks, but we didn't receive any contributions regarding the pluggability for SD.

As seen in the past, providing a way to disable a functionality, just encourages more wrappers and that results in fragmentation & impacts the user experience badly. But, having a pluggable implementation gives the correct balance of functionality and user experience. We are seeing that in action in Volumes and network plugins already.

am highly in favor of a pluggable SD approach and any contributions to that is welcome.

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Jan 20, 2016

Contributor

Closing, sounds like the answer is "You don't" 😄

Contributor

ibuildthecloud commented Jan 20, 2016

Closing, sounds like the answer is "You don't" 😄

@kevinxucs

This comment has been minimized.

Show comment
Hide comment
@kevinxucs

kevinxucs Apr 12, 2016

Seeing random dns resolve errors with macvlan, an option to disable embedded dns would be greatly appreciated.

kevinxucs commented Apr 12, 2016

Seeing random dns resolve errors with macvlan, an option to disable embedded dns would be greatly appreciated.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Apr 12, 2016

@kevinxucs Are the errors for docker service names or external names ? Can you also run the daemon in the debug mode and share the logs when the resolution failure happens ?

If you are using 1.11 images update to the latest RC5 image; which has some bug fixes related to DNS resolution.

sanimej commented Apr 12, 2016

@kevinxucs Are the errors for docker service names or external names ? Can you also run the daemon in the debug mode and share the logs when the resolution failure happens ?

If you are using 1.11 images update to the latest RC5 image; which has some bug fixes related to DNS resolution.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 12, 2016

Member

@kevinxucs also, please open a separate issue with the requested information

Member

thaJeztah commented Apr 12, 2016

@kevinxucs also, please open a separate issue with the requested information

@bboreham

This comment has been minimized.

Show comment
Hide comment
@bboreham

bboreham Apr 13, 2016

Contributor

I started to ponder the idea of a "DNS plugin" for Docker. It raised some fundamental questions that I would like to get some reaction to, in advance of making a full proposal.

I work on Weave; my primary concern would be to answer look-ups for containers attached to a Weave network. But containers may be attached to multiple networks, and DNS doesn't support the concept of multiple servers that know about different sub-spaces, so our plugin has to be able to answer about all containers attached to all networks.

Assuming the previous point holds, where would our plugin get this information from? Practically, it needs access to the same data that Docker DNS is currently using. It could be fed this via a subscription mechanism, with some way to re-sync on restart.

On the query side, what are the requirements that a Docker DNS plugin must meet? For instance, does it have to preserve the behaviour that only container names are used for resolution, not hostnames? Does it have to answer reverse lookups with container_name.network_name? Those are two key things I would want to change, so if they are fixed-points in the requirements I don't need to start down this road.

Contributor

bboreham commented Apr 13, 2016

I started to ponder the idea of a "DNS plugin" for Docker. It raised some fundamental questions that I would like to get some reaction to, in advance of making a full proposal.

I work on Weave; my primary concern would be to answer look-ups for containers attached to a Weave network. But containers may be attached to multiple networks, and DNS doesn't support the concept of multiple servers that know about different sub-spaces, so our plugin has to be able to answer about all containers attached to all networks.

Assuming the previous point holds, where would our plugin get this information from? Practically, it needs access to the same data that Docker DNS is currently using. It could be fed this via a subscription mechanism, with some way to re-sync on restart.

On the query side, what are the requirements that a Docker DNS plugin must meet? For instance, does it have to preserve the behaviour that only container names are used for resolution, not hostnames? Does it have to answer reverse lookups with container_name.network_name? Those are two key things I would want to change, so if they are fixed-points in the requirements I don't need to start down this road.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Apr 13, 2016

@bboreham We have considered the possibility of having DNS plugins. One of the challenges is that the DNS plugin being a container level entity should be able to resolve the service names for any network the container is connected to. But the service abstraction (service naming, search domain etc.) provided by different vendors (k8s, weave etc.) may not be the same.

A better alternative is to define a schema for service naming that works for majority of the vendors (things like how container name vs host name should be used, what should be the domain name etc.). This might be cleaner approach to address the service naming/discovery in a consistent way across vendors and the Docker inbuilt drivers.

sanimej commented Apr 13, 2016

@bboreham We have considered the possibility of having DNS plugins. One of the challenges is that the DNS plugin being a container level entity should be able to resolve the service names for any network the container is connected to. But the service abstraction (service naming, search domain etc.) provided by different vendors (k8s, weave etc.) may not be the same.

A better alternative is to define a schema for service naming that works for majority of the vendors (things like how container name vs host name should be used, what should be the domain name etc.). This might be cleaner approach to address the service naming/discovery in a consistent way across vendors and the Docker inbuilt drivers.

@bboreham

This comment has been minimized.

Show comment
Hide comment
@bboreham

bboreham Apr 13, 2016

Contributor

Sorry, I picked up on

make it pluggable in this release. Contributions welcome.

having a pluggable implementation gives the correct balance of functionality and user experience

it is about making it pluggable

My bad.

Contributor

bboreham commented Apr 13, 2016

Sorry, I picked up on

make it pluggable in this release. Contributions welcome.

having a pluggable implementation gives the correct balance of functionality and user experience

it is about making it pluggable

My bad.

@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Apr 13, 2016

There might be use cases that are better addressed by a pluggable DNS. I am not disagreeing on that. But without a consistent schema it makes it difficult to implement a pluggable DNS that can provide the same functionality for all networks the container is connected to.

sanimej commented Apr 13, 2016

There might be use cases that are better addressed by a pluggable DNS. I am not disagreeing on that. But without a consistent schema it makes it difficult to implement a pluggable DNS that can provide the same functionality for all networks the container is connected to.

@wrouesnel

This comment has been minimized.

Show comment
Hide comment
@wrouesnel

wrouesnel Jan 17, 2017

Contributor

So the --dns option doesn't seem to do anything the documentation implies - using it it looks like docker is just eating up the option and using the internal DNS server (in 1.12.1) to try and resolve the name...which fails, because the DNS server is only reachable via the network plugin I'm using, and not via my hosts network.

This seems like the culmination of ongoing insanity: there's no parameter to tell docker "don't bind mount resolv.conf", and now there's no option to say "just put this content in resolv.conf". Instead we've got docker trying to be a DNS server and failing at it.

Contributor

wrouesnel commented Jan 17, 2017

So the --dns option doesn't seem to do anything the documentation implies - using it it looks like docker is just eating up the option and using the internal DNS server (in 1.12.1) to try and resolve the name...which fails, because the DNS server is only reachable via the network plugin I'm using, and not via my hosts network.

This seems like the culmination of ongoing insanity: there's no parameter to tell docker "don't bind mount resolv.conf", and now there's no option to say "just put this content in resolv.conf". Instead we've got docker trying to be a DNS server and failing at it.

@antoineco

This comment has been minimized.

Show comment
Hide comment
@antoineco

antoineco Jan 31, 2017

This is still creating a lot of issues as of Docker 1.13, especially with Docker Compose ignoring the dns parameter (ref. docker/compose#3078) and generating invalid host names (ref. docker/compose#229).

DNS resolution from some container started with dns: 8.8.8.8 (docker-compose):

:/# cat /etc/resolv.conf
nameserver 127.0.0.11    # <-- wat
options ndots:0

:/# nslookup 172.18.0.3
Name:      172.18.0.3
Address 1: 172.18.0.3 modcluster_lb_1.modcluster_default    # <-- wat

And the consequence on a client. Here an Apache httpd server:

[core:debug] [pid 280:tid 139910987741872] vhost.c(796): [client 172.18.0.4:57214] AH02415: [strict] Invalid host name 'modcluster_lb_1.modcluster_default', problem near: _lb_
[core:debug] [pid 280:tid 139910987741872] vhost.c(891): [client 172.18.0.4:57214] AH00550: Client sent malformed Host header: modcluster_lb_1.modcluster_default:6666
[core:debug] [pid 280:tid 139910987741872] protocol.c(1385): [client 172.18.0.4:57214] AH00569: client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /

antoineco commented Jan 31, 2017

This is still creating a lot of issues as of Docker 1.13, especially with Docker Compose ignoring the dns parameter (ref. docker/compose#3078) and generating invalid host names (ref. docker/compose#229).

DNS resolution from some container started with dns: 8.8.8.8 (docker-compose):

:/# cat /etc/resolv.conf
nameserver 127.0.0.11    # <-- wat
options ndots:0

:/# nslookup 172.18.0.3
Name:      172.18.0.3
Address 1: 172.18.0.3 modcluster_lb_1.modcluster_default    # <-- wat

And the consequence on a client. Here an Apache httpd server:

[core:debug] [pid 280:tid 139910987741872] vhost.c(796): [client 172.18.0.4:57214] AH02415: [strict] Invalid host name 'modcluster_lb_1.modcluster_default', problem near: _lb_
[core:debug] [pid 280:tid 139910987741872] vhost.c(891): [client 172.18.0.4:57214] AH00550: Client sent malformed Host header: modcluster_lb_1.modcluster_default:6666
[core:debug] [pid 280:tid 139910987741872] protocol.c(1385): [client 172.18.0.4:57214] AH00569: client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jan 31, 2017

Member

The embedded nameserver acts as a forwarder to the dns you specified with the --dns flag, so you'll always see 127.0.0.11 as DNS. Docker compose using underscores is a separate issue (although you can override the hostname for a container)

Member

thaJeztah commented Jan 31, 2017

The embedded nameserver acts as a forwarder to the dns you specified with the --dns flag, so you'll always see 127.0.0.11 as DNS. Docker compose using underscores is a separate issue (although you can override the hostname for a container)

@antoineco

This comment has been minimized.

Show comment
Hide comment
@antoineco

antoineco Jan 31, 2017

@thaJeztah thanks for your answer.
I perfectly understand how things work underneath and why they were done that way, however if I am explicit with the DNS server I want to use (or don't) I don't see why Docker would forcibly respond to my DNS queries with a hostname I don't expect.

The hostname of a container can be set, yes, but in the context of Compose the Docker DNS ignores it completely, which doesn't solve the original problem:

❯ docker exec modcluster_lb_1 hostname
lb
❯ docker exec modcluster_lb_1 hostname -i
172.18.0.2
❯ docker exec modcluster_backend_1 nslookup 172.18.0.2
Name:      172.18.0.2
Address 1: 172.18.0.2 modcluster_lb_1.modcluster_default  # <- nope. thanks

Compose may be the source of the underscores, but it wouldn't be such a big deal if I could bypass the DNS.

antoineco commented Jan 31, 2017

@thaJeztah thanks for your answer.
I perfectly understand how things work underneath and why they were done that way, however if I am explicit with the DNS server I want to use (or don't) I don't see why Docker would forcibly respond to my DNS queries with a hostname I don't expect.

The hostname of a container can be set, yes, but in the context of Compose the Docker DNS ignores it completely, which doesn't solve the original problem:

❯ docker exec modcluster_lb_1 hostname
lb
❯ docker exec modcluster_lb_1 hostname -i
172.18.0.2
❯ docker exec modcluster_backend_1 nslookup 172.18.0.2
Name:      172.18.0.2
Address 1: 172.18.0.2 modcluster_lb_1.modcluster_default  # <- nope. thanks

Compose may be the source of the underscores, but it wouldn't be such a big deal if I could bypass the DNS.

@lierdakil

This comment has been minimized.

Show comment
Hide comment
@lierdakil

lierdakil Feb 25, 2017

FWIW, I'm running into issues with embedded DNS on a host that uses nftables instead of iptables (iptables are disabled because that conflicts with nftables' dnat). It just doesn't work, plain and simple, for obvious reasons. While being able to disable eDNS won't solve service discovery problem, I could work around that.

lierdakil commented Feb 25, 2017

FWIW, I'm running into issues with embedded DNS on a host that uses nftables instead of iptables (iptables are disabled because that conflicts with nftables' dnat). It just doesn't work, plain and simple, for obvious reasons. While being able to disable eDNS won't solve service discovery problem, I could work around that.

@valentin2105

This comment has been minimized.

Show comment
Hide comment
@valentin2105

valentin2105 Jul 18, 2017

Hi all,

I try to use Docker with Docker-compose in IPv6 only (so in user definited network) and I couldn't make it work due of the embedded DNS. If I pass the --dns flag, it doesn't work because my host don't have any IPv4 connectivity.

Any plan to allow disable embedded DNS in the future ? (for this kind of use case) ?

valentin2105 commented Jul 18, 2017

Hi all,

I try to use Docker with Docker-compose in IPv6 only (so in user definited network) and I couldn't make it work due of the embedded DNS. If I pass the --dns flag, it doesn't work because my host don't have any IPv4 connectivity.

Any plan to allow disable embedded DNS in the future ? (for this kind of use case) ?

sebschrader added a commit to agdsn/hades that referenced this issue Aug 5, 2017

Use default bridge network
Using Docker's default bridge network disables the embedded DNS server
listening on 127.0.0.11 inside the container. This seems to be the only the way
to disable the embedded DNS server, according to:
moby/moby#19474
@christopherobin

This comment has been minimized.

Show comment
Hide comment
@christopherobin

christopherobin Aug 24, 2017

To add to the discussion, I wish I could opt-out some networks from the resolution. I am currently running a swarm internally where developers can run their copy of a web app for testing purposes.

Load balancing is done via Traefik running as a docker service, containers join the traefik network and use labels to declare how they want to be exposed.

The issue with the internal DNS is that 2 copies of app "foobar" might be started via docker stack, they join 2 networks: <stack_name>_default and traefik. Then the app tries to connect to the database host and... actually connect to each other's database through the traefik network randomly due to them being both on the traefik network.

Being able to say "this network doesn't allow for resolution" would solve my issue immediately. Right now my only solutions is either to use network aliases with environment variables (like inserting deploy IDs in the aliases), or to move Traefik out and instead write some sort of external registrator that watches the swarm and create entries in consul.

christopherobin commented Aug 24, 2017

To add to the discussion, I wish I could opt-out some networks from the resolution. I am currently running a swarm internally where developers can run their copy of a web app for testing purposes.

Load balancing is done via Traefik running as a docker service, containers join the traefik network and use labels to declare how they want to be exposed.

The issue with the internal DNS is that 2 copies of app "foobar" might be started via docker stack, they join 2 networks: <stack_name>_default and traefik. Then the app tries to connect to the database host and... actually connect to each other's database through the traefik network randomly due to them being both on the traefik network.

Being able to say "this network doesn't allow for resolution" would solve my issue immediately. Right now my only solutions is either to use network aliases with environment variables (like inserting deploy IDs in the aliases), or to move Traefik out and instead write some sort of external registrator that watches the swarm and create entries in consul.

@matthiasdg

This comment has been minimized.

Show comment
Hide comment
@matthiasdg

matthiasdg Oct 9, 2017

@christopherobin I am experiencing the same issue in docker compose. Only seeing it when the project's default network is alphabetically following the traefik network, though, corresponding to https://docs.docker.com/engine/userguide/networking/

When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order

Was also thinking about using network aliases as a fix

matthiasdg commented Oct 9, 2017

@christopherobin I am experiencing the same issue in docker compose. Only seeing it when the project's default network is alphabetically following the traefik network, though, corresponding to https://docs.docker.com/engine/userguide/networking/

When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order

Was also thinking about using network aliases as a fix

@PaulFurtado

This comment has been minimized.

Show comment
Hide comment
@PaulFurtado

PaulFurtado Oct 9, 2017

We are transitioning to use a user-defined bridge so we can use our own IPAM driver since the default docker bridge does not allow you to use a custom IPAM driver. Each of our servers runs 400+ containers, and redundant caching DNS servers on each node. Some of our services are quite heavy on DNS - individual containers can push 60k+ DNS QPS at times, making even bind and dnsmasq CPU-bound; we really can't rely on the built-in docker DNS server. It's disappointing to see docker remain adamant against adding a flag to disable this for user-defined networks when it's on such a critical path, especially since we don't use docker's service discovery features in our infrastructure. We'll have to work around this by adding another patch to our docker build. What is so special about user-defined networks that makes them require this DNS server to be enabled when the default bridge does not? Looking at the code it seems like it'd be very trivial to add such a flag.

PaulFurtado commented Oct 9, 2017

We are transitioning to use a user-defined bridge so we can use our own IPAM driver since the default docker bridge does not allow you to use a custom IPAM driver. Each of our servers runs 400+ containers, and redundant caching DNS servers on each node. Some of our services are quite heavy on DNS - individual containers can push 60k+ DNS QPS at times, making even bind and dnsmasq CPU-bound; we really can't rely on the built-in docker DNS server. It's disappointing to see docker remain adamant against adding a flag to disable this for user-defined networks when it's on such a critical path, especially since we don't use docker's service discovery features in our infrastructure. We'll have to work around this by adding another patch to our docker build. What is so special about user-defined networks that makes them require this DNS server to be enabled when the default bridge does not? Looking at the code it seems like it'd be very trivial to add such a flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment