Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Containers cannot resolve DNS if docker host uses 127.0.0.1 as resolver #541
Comments
|
The kernel segfault is a known bug with kernels < 3.8 (cf #407). Regarding the internet access, can you look at your /etc/resolv.conf? It is most likely setup on a localhost. This works within the host but can't work within a container. 2 solutions:
|
|
Maybe we should detect when /etc/resolv.conf is configured to use 127.*, On Tue, May 7, 2013 at 10:00 AM, Guillaume J. Charmes <
|
|
Perhaps we could use the proxy from docker to proxy these requests to localhost? e.g.: use 172.16.42.1 within docker containers as the DNS server and use 127.0.0.1 as the real DNS |
|
Well, IMHO, if we detect that the DNS server is 127.*, we should generate a We had lots of discussions with @creack a long time ago about this, so I |
|
@jpetazzo Unfortunately dnsmasq isn't set up by default to accept requests on the IP of the docker0 bridge, even though it seems to be bound to 0.0.0.0. I've just verified whether dnsmasq allows DNS resolution when using the docker0 bridge IP address as the DNS server. Setting it up isn't an option, imo. We could set up docker to look for an alternative resolv.conf file (e.g. /etc/resolv.docker). If 127.* is in resolv.conf, no resolv.docker exists and no -dns option was passed to docker -d or docker run, we could use public DNS and print a warning on the screen (!!! WARNING - docker is using public DNS because /etc/resolv.conf lists 127.0.0.1 as a DNS server). |
|
Agreed! On Tue, May 7, 2013 at 12:44 PM, unclejack notifications@github.com wrote:
|
This was referenced May 16, 2013
To whatever extent possible I think containers shouldn't know about Docker, so injecting Docker-specific knowledge inside the container is something to avoid. If I misunderstood you and you were instead proposing having the /etc/resolv.conf be outside the containers with a fallback to hardcoded public DNS (but where |
|
@fj It was about the /etc/resolv.conf and /etc/resolv.docker which would live on the host. Docker will still manage everything on its own when setting up the resolv.conf in the container, it would just pick a DNS using the rules I've described. |
This was referenced May 31, 2013
|
This is fixed in master. The docker daemon will detect that the host's resolv.conf points to localhost, and fallback to a public default (8.8.8.8). This can then be overridden by the admin with 'docker -d -dns'. |
shykes
closed this
Jun 14, 2013
mlebkowski
commented
Nov 29, 2013
|
It seems that it’s not just the case of localhost.
It seems that docker uses the same network as the host. I can’t really tell if this is a docker issue or my network configuration. Should I reopen the ticket? |
|
Well, those networks are indeed overlapping, but that looks like it might be a separate issue, since Docker's automatic network detection code should've avoided exactly that, as far as I remember (it's been a while since I looked at that part of the code). I believe the exact DNS fix described here was that if your DNS server is anything in a "local" block, it gets remapped to 8.8.8.8, so even in your case it should've been remapped, as far as my understanding goes. Perhaps someone with more knowledge of that specific part of the code will chime in here. :) |
cattz
commented
Dec 10, 2013
|
Maybe a better option, other than remap the nameserver to a public one would be to have an way to configure the 'default' nameserver for containers. I can get DNS working fine when I run it with the |
cattz
commented
Dec 10, 2013
|
I can reply to my own question: dotcloud#815
|
anentropic
commented
Dec 28, 2013
|
Puzzled newbie here... I am running Docker inside of a Vagrant vm Yesterday I was able to I tried adding My
What am I supposed to do to make it work? |
anentropic
commented
Dec 28, 2013
|
and... rebooting the vagrant vm fixed it |
The 'fix' above excludes all production environments which deliberately run a caching nameserver on 127.0.0.1. Our resolv.conf reads:
Is there a way to support environments which run a per-host caching-only dns recursive resolver? |
BadgerOps
commented
Jan 15, 2014
|
I'm experiencing the exact same issue as tamsky - My company does a caching nameserver at 127.0.0.1 and my resolv.conf reads the same as tamsky's. I've appended 'nameserver 8.8.8.8' to my resolv.conf, but it appears docker never checks that far down? In my honest opinion, this kills the portability factor of docker, as currently the daemon has to be restarted using a specified dns server. It looks like there have been several issue threads discussing this problem - what is the possibility/probability of a docker.conf where we can specify instance variables like dns? Thanks! |
|
@badger32d just as a point of interest, if all you need to "fix" this in your environment is a configuration file to specify Just put something like this inside: DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4"Then So it's not exactly a solution to all the issues, but it's certainly a reasonable way to specify the workaround in a simple and persistent way. |
BadgerOps
commented
Jan 16, 2014
|
Tianon: I didn't realize that was a possibility, thanks! That should solve it. |
|
@badger32d 's solution will not work on our network. I'm not sure what our workaround will be but it might involve having named listen on the docker0 interface, and setting the DOCKER_OPTS="-dns 172.17.42.1" |
|
@tamsky yes, you need to have your resolver bind on the docker0 bridge. Then pass the -dns flag to the docker daemon with the bridge ip. This ill make all container's created use the bridge as the resolver. |
|
It sure would be nice if these issues could be addressed:
Can we, as part of our system networking startup configuration, create and configure the bridge interface before dockerd starts? If so, that would resolve both issues. |
|
@tamsky Absolutely! If you create/configure the docker0 interface before starting Docker, Docker will pick up the network settings from the interface instead of making them up (for example, using the IP/mask on the bridge instead of trying to find a suitable one). You can also use $ docker -h
Usage of docker:
...
-b, --bridge="": Attach containers to a pre-existing network bridge; use 'none' to disable container networking
--bip="": Use this CIDR notation address for the network bridge's IP, not compatible with -b
... |
seshendra
commented
Feb 25, 2014
|
I have installed docker on 1) Ubuntu 12.04 and 2) CentOS 6.4. I did this on my work laptop connected to our corporate VPN, as I was trying to demonstrate docker for a potential production use. However, I cannot connect to internet (if I had to run a Do you guys think it is related to the DNS issue? |
dodev
commented
Feb 25, 2014
|
@seshendra I think I'm facing the same problem:
And yes, I think it's DNS related :) |
seshendra
commented
Mar 6, 2014
|
@dodev for now I solved this by doing two things
I'm not exactly sure which one solved the DNS issue --- but working for now on CentOS. Haven't tried on Ubuntu yet. |
shafi-codez
commented
Mar 18, 2014
|
@seshendra : Did you add any changes to /etc/resolv.conf file or set any env variable for this? |
seshendra
commented
Mar 19, 2014
|
@shafi - I've upgraded to docker 0.9 and everything seems to be working fine both on centos and Ubuntu. Thanks!
|
wt
commented
May 16, 2014
|
I am going to use this trick on the docker host:
The 192.168.0.1 is the docker default gateway. You can now do "nc 192.168.0.1 25" to get the mail server. The same trick can be probably be used for DNS. You'll have to use "-p udp" instead, of course. I haven't tried this with DNS, so YMMV. And i guess you could take it one step further and add a DNAT rule inside the container to direct requests to 127.0.0.1:25 to 192.168.0.1:25 if you are hell bent on having the container talk to 127.0.0.1 instead of 192.168.0.1. |
liamzebedee
referenced this issue
in docker/compose
Jul 28, 2014
Closed
VERY SLOW network connections to localhost container when not connected to Internet #356
Quasaur
commented
Aug 14, 2014
|
on Debian 7.6 i have the same issue... Generated by Connection Managernameserver 209.18.47.61 do i need bind9 (or some something slimmer) installed to resolve addresses for docker? If so, why doesn't everything else on my system seem to resolve addresses with no issue? |
|
@Quasaur containers can't reach that 127.0.0.1 address (since they have their own 127.0.0.1). |
bjornpost
referenced this issue
in dokku/dokku
Sep 4, 2014
Closed
Problems pushing anything, and updating build step – Ubuntu trusty? #680
seemant
commented
Oct 30, 2014
|
I am having same issue.. Trying to set up 3 docker instances for a test-automation. Each instance has a local IP. The automation test-bed assumes that there are 3 machines with given IP. It doesn't care if the 3 machines are dockers or VM instances or physical machines. So we do not want to use docker commands to set up the machines for automation. We create a base docker ubuntu image with SSH service enabled. Then we use SSH commands in the robotframework to set up the machines with required packages. Along the way we need to setup /etc/hosts file so that apps running on one machine can discover the apps on the other. This used to work when we started using docker first time.. With recent changes its broken and we cannot edit the /etc/hosts file. So for work around I thought I would try to install dnsmasq and change the /etc/resolv.conf file to point to 127.0.0.1, then with dnsmasq options I could specify some other file for static host name to ip bindings. While this works if the apps inside the docker explicitly send query to 127.0.0.1, it doesn't work when apps rely on gethostbyname API(s) provided by the system. So, I thought of modifying resolv.conf then realized that .. it cannot be modified and it set to Google DNS servers 8.8.8.8 and 8.8.4.4. I thought we can modify the resolvconf "head" file and restart resolvconf service.. But wait.. the service complains that the /etc/resolv.conf is not a link to /var/run/resolvconf/resolve.conf. So there goes that approach out the window. So now what is left.. After some more googling I find this solution here: Brilliant minds here suggest to use sudo docker -d dns to change the DNS server of a a running docker instance.. That means automation tests need to have 2 approaches, now one for docker based test-bed and one for VM based test-bed.. We can do that I guess. but I am not seeing the point of all these restrictions in a docker? If its security that is bothering,, then can we have an option to build a "non-secure" image that allows changes to anything? Thanks |
|
I experienced the same error today when I switched WiFi networks. I was able to fix this with |
rektide
commented
May 27, 2015
|
I flit between some essential internal DNS servers, and am on Ubuntu. With these tweaks, their dnsmasq set-up & run by their network-manager serves my Docker containers-
|
haugene
referenced this issue
in haugene/docker-transmission-openvpn
May 28, 2015
Closed
Self-built docker unable to resolve host #19
jonseymour
commented
Jul 1, 2015
|
I also experienced the issue mcandre reported on OSX when switching from WiFi to 3G after starting boot2docker on WiFi. The suggested workaround |
chrisconley
commented
Jul 24, 2015
|
I was also able to fix this issue with |
This was referenced Aug 3, 2015
mmahalwy
commented
Dec 7, 2015
|
|
infosatheesh2020
commented
Feb 8, 2016
|
My host /etc/resolv.conf points to 192.168.1.1, and I have set dnsmasq on docker0 (172.17.42.1) interface to use local dns resolver for containers (similar to http://docs.blowb.org/setup-host/dnsmasq.html). now, in host machine, internet dns queries are resolved in both 192.168.1.1 and 172.17.42.1 interfaces. I have to use --dns 172.17.42.1 option to docker containers, since dnsmasq on docker0 hosts local dns entries of other containers. But the problem here is that public URLs archive.ubuntu.com is not resolving and so package update fails inside container Docker Host: (/etc/resolv.conf) Inside container: (/etc/resolv.conf) Is there any workaround for this case |
tayzlor
referenced this issue
in Capgemini/Apollo
Feb 18, 2016
Closed
Registrator fails on reboot #617
roelvanduijnhoven
commented
Apr 1, 2016
|
Why can't Docker figure out my DNS is configured to We rely on some special DNS rules on our own DNS server. The PC points to that DNS server, however because /etc/resolv.conf contains 127.0.0.1, it will default to Google DNS. So our DNS rules don't work and stuff breaks during development. |
|
@roelvanduijnhoven as mentioned here : docker#541 (comment) this is the expected behavior when /etc/resolv.conf contains 127.0.0.1. But you can use |
roelvanduijnhoven
commented
Apr 2, 2016
|
That does work, I already fixed it this way on my machine. But this is not ideal. You will need to reconfigure this setting every time your laptop moves to another location as the DNS location will change. |
raarts
commented
Apr 30, 2016
|
Maybe this is useful to others, I solved it as follows:
This makes both regular containers as well as those on user-defined networks just work. |
|
I understand that it's desirable to rewrite /etc/resolv.conf if it only has 127.0.0.1 listed because the containers are isolated in their own network. But if I'm using net="host" isn't this behavior a bug? Specifically I think I'm seeing a race because one container get's started with the correct /etc/resolv.conf from the host and the other one (also net:"host") get's a rewritten /etc/resolv.conf with public dns's listed. |
|
There is an outstanding issue to change this behaviour in the case of |
|
Also related: docker#22408 |
skirdey
commented
Jun 10, 2016
|
This helped in my case: config.vm.provision :shell, :inline => "echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf &> /dev/null", :privileged => true |
flypenguin
commented
Jul 7, 2016
|
I have just hit this issue. Hard. |
|
@flypenguin please upgrade to 1.11 and try again. Thanks! |
flypenguin
commented
Jul 7, 2016
|
Hm. Tested on 1.9.1 and 1.11.2, didnt work in both ... |
sanimej
commented
Jul 7, 2016
|
@flypenguin Yes, forwarding the queries to the host's 127.0.0.1 is not supported now. You can use the --dns option on container run. Will that work for you ? |
flypenguin
commented
Jul 11, 2016
•
|
[Update] Okay, solved it it seems. Ubuntu 16.04 installed - for whatever reason - the lxc packages, which contain a local dnsmasq set up which somehow prevents DNS to be used in the containers. Seems I have to read up on what that is, but uninstalling lxc* packages did the trick. Super weird. [Original post] It might - but I think I hit another problem. I opened all DNS servers on the "public" network interfaces now, and on some machines it "just works", on others it doesn't. It actually seems to be distribution dependent. On Ubuntu 16.04:
On Ubuntu 15.10:
Summary:
Both hosts should have exactly the same configuration (I use the same puppet rules for both), and I don't know of any difference except some installed software or IP address out of my head. |
|
This might be related with the issue I was seeing and posted above but was not really given much attention. Namely, I think I was seeing a race condition because the issue was not happening on different OS version but on the same OS in different containers. Some got the correct /etc/resolv.conf some did not. I have not checked with the latest release. EDIT: I did some quick testing on 1.11.2: I ran the command a couple of times and I can't seem to reproduce like I was able before. So at least the perceived race is gone. |
rage-shadowman
commented
Sep 13, 2016
|
Will there be a real fix to this? Such that if /etc/resolve.conf points to localhost, the docker daemon will forward dns requests to localhost? (as opposed to forwarding them to google's dns servers) |
|
@rage-shadowman Are you talking with "--net=host"? Cause that appears to to be working with 1.11.2. If you're not talking about "--net=host"... I don't see how that could work. |
rage-shadowman
commented
Sep 14, 2016
•
|
No, I mean using the default bridge network. You are already filtering out localhost from /etc/resolv.conf. Why not simply replace it with the ip address of the host? My current hack is to do this in my .bashrc:
However, this is an ugly hack with its own issues. It currently works for me in my dev environment but a real solution which points dns requests to the host machine would be much preferable. PS - I will admit to having very little knowledge of the inner workings of docker. However, to my uneducated mind this seems like something that should be workable. |
|
Also see docker#23910 |
margh
commented
Jan 5, 2017
|
FWIW On Ubuntu 15 with a Docker seemed to be clipping |
moepot
commented
Jan 25, 2017
|
I agree with @rage-shadowman and @flypenguin . There should be a better solution than to simply point to the Google nameservers, because one wants to use his own network's nameserver. Why is this issue closed? Are you going to fix this problem? |
|
@moepot you can use your own network's name server; the problem in this case is that a container has it's own network namespace; 127.0.0.1 is localhost inside the container, not localhost of the host that the container runs on. Other IP-addresses on the local network should work. |
flypenguin
commented
Jan 25, 2017
|
I think everybody is aware why that is not working. I also think most people here (definitely including myself) think docker should just make this possible ;) |
moepot
commented
Jan 25, 2017
|
@thaJeztah thanks for your answer. I understand the technical details, but I'm not very happy with the current solution. Of course I can set my own name server with the So my question is: Do we have to live with the |
sanimej
commented
Jan 25, 2017
|
@moepot @flypenguin @margh @thaJeztah Fix for this is available in docker master and will be included in 1.14. With Docker DNS server we can differentiate between host's loopback vs container's loopback (passed though |
|
@sanimej do you have the PR that vendored it into docker? |
sanimej
commented
Jan 25, 2017
|
@thaJeztah Its docker#29891. With this change there is an issue in upgrading from 1.12.x to 1.14. Its fixed in libnetwork. I will vendor it shortly. |
referenced
this issue
in volumio/Build
Mar 3, 2017
tundratim
commented
Aug 31, 2017
|
Have just installed docker-engine 17.06.1-ce build 874a737 Same problem. Running a host (VMware) instance of dnsmasq that listens on 127.0.0.1 and resolv.conf reflects this. Docker containers cannot resolve through this server. I was under the impression that this had been fixed. Ideas welcome. One possible thought here is to alias a second, non-routable IP onto the host NIC and run the dnsmasq instance there, thereby giving it a non-localhost listening address. Has anyone tried this? |
themotu
commented
Dec 14, 2017
|
for me containers start and have working resolution, after random amount of time all dns resolution fails |
jbwyatt4 commentedMay 7, 2013
I'm trying to install rails on a docker container. However the OS inside docker cannot seem to connect to the internet. I get this error message on the base (just apt-get wasn't able to connect to the internet).
sudo: unable to resolve host e62b9fa31bb9
Err http://archive.ubuntu.com quantal InRelease
Err http://archive.ubuntu.com quantal Release.gpg
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/InRelease
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/Release.gpg Temporary failure resolving 'archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
I've had this issue since docker 0.1 to 0.3. Running on either the host machine or in vmware. (Virtualbox does not have this issue but sometimes the kernel segfaults within the container.)