New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't access internet from containers #13381
Comments
@VirtualSniper do you have net.ipv4.ip_forward on? |
yes |
@VirtualSniper are you able to reproduce this ? |
@aboch yes, here is : |
I was also running into this. I was able to work around it by deactivating & deleting docker0 (after stopping docker), and then starting docker again (which re-creates docker0). I do have NetworkManager running, since this is on my laptop. and a VPN running at the moment. But this was also happening while I was in the office on Friday (sans VPN in use). I wouldn't be surprised is this is a known issue, and there are other open issues for it. Anyone know who to tag from the Docker networking side to find out more? |
I have the same problem but it not always happens. First, everything works fine, then after few days passed, or I build some images, start/stop containers. Sometimes inside the container cannot connect to anything anymore. All my running containers are the same, lost internet connection, for example curl to github (via ip or domain) will fail:
The only way I can solve this is restart docker daemon, then everything will be back to work. Any suggestions for this? Thanks. |
ping @aboch |
There was a bug in bridge driver code where the linux bridge interface MAC address would not be programmed as "SET" in 4.x (x < 3) kernels. Bug is there in docker 1.6.2. @cfpeng I see your host is running a 4.0.2, so it would be affected. @bear0330 can you check whether your kernel is a 4.x as well ? This would explain why you hit the issue after a while, maybe after spawning a new container. @cfpeng @bear0330 could you please check whether you are still hitting this issue with latest 1.8.0-rcX image |
@aboch My kernel is 3.10.0-229.7.2.el7.x86_64, I am running docker on Azure, I am not sure this is because Azure's issue (I have no idea), I am trying to run docker on Vultr. |
Similar issue. Removing the bridge (and relevant cleanup) did not have an effect. $ uname -a $ docker version Server: $ docker run -ti ubuntu /bin/bash $ ping www.google.com |
@bear0330 Your issue (from what I can see from your logs) is different from the one hit by @cfpeng and @fkeet. Theirs is related to DNS response packets not being delivered to the container, yours is related to ip reachability: you would get "no route to host" if ,for example, in your container the default gw IP is unset or it does not belong to the same network of eth0. |
@cfpeng Given DNS requests are routed from docker0 to eth0 but responses are not, it makes me think it has to do with iptables. @fkeet Can you also try the same. Thanks. |
@aboch I got the same issue after running docker on Vultr's machine for few days. Now my container cannot connect to internet again. Now in container (My container's hostname is status.xxx.com):
(hang, I press Ctrl+C to break)
run
(hang..., also Ctrl+C)
There is some log messages in my /var/log/messages, I don't know it is related or not:
If you need any further information, please tell me I would provide it if I can. |
@aboch I have upgrade to 1.8.1, the issue is still exist. |
/cc @LK4D4 |
@aboch Here is:
Chain INPUT (policy ACCEPT 1929 packets, 117K bytes) Chain OUTPUT (policy ACCEPT 1126 packets, 69647 bytes) Chain POSTROUTING (policy ACCEPT 1119 packets, 69150 bytes) Chain DOCKER (2 references)
Generated by resolvconfnameserver 8.8.8.8 |
@cfpeng do you have selinux? |
@aanand no. |
I am encountering this issue too. I have to systemctl restart docker on Arch Linux to get access to the internet from within containers. |
I'm running into the same problem, tried everything found on google but nothing fixed the issue. $ sudo iptables -t nat -L -nv Chain INPUT (policy ACCEPT 6 packets, 423 bytes) Chain OUTPUT (policy ACCEPT 1132 packets, 128K bytes) Chain POSTROUTING (policy ACCEPT 1132 packets, 128K bytes) Chain DOCKER (2 references) $ sudo docker run --rm -it ubuntu /bin/bash |
After I reinstall the OS, the problem is resolved. |
I had same error. |
I've encountered a similar issue. Containers can't access the internet until a manual restart One thing I've noticed is when my computer just boot up. Here's the output before restarting docker:
After docker restarted:
not sure if this would help. |
@aboch Thx for the feedback. How can i do that exactly? |
While the ping command is running in your container check the following:
If you don't see the expected tx/rx after each of the steps above, check repeatedly the iptables filter rules on another shell: |
Thx @aboch for your help. Here is my result: i can see the outgoing packets on the interfaces, but no reply:
23:23:48.076002 IP (tos 0x0, ttl 64, id 29434, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 9984, seq 1118, length 64
23:23:43.074293 IP (tos 0x0, ttl 63, id 28889, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 9984, seq 1113, length 64
23:24:26.092552 IP (tos 0x0, ttl 64, id 35006, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 9984, seq 1156, length 64 looking at the iptables counts, the only counter which changes is: Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8 392 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0 /* 100v4: forward to DOCKER chain */
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 /* 101v4: accept docker ESTABLISHED,RELATED */ state RELATED,ESTABLISHED
# --- following rules is counting up!
49721 2776K ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 /* 102v4: accept
# ---
docker0 outgoing */*
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0 /* 103v4: accept docker0 local */
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 810v4-drop: drop forward */ reject-with icmp-port-unreachable
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 950v4-basicfw: drop forward */ reject-with icmp-port-unreachable i can't see another rule with raising counter which might drop the packets. could it somehow be related to the missing |
Thanks for the info. |
@aboch the problem is somehow related to the docker setup itself and not caused by problem from outside the host: as soon as i restarted the docker service, the network issue dissapears. |
@aboch @svenmueller after digging a little bit more, I found out that the NAT rule in the POSTROUTING chain is missing whenever the "Gateway" key is not in the output of the "docker network inspect bridge" command. |
@aboch I want to add that I have this issue on my local install too, using docker-machine (Mac OS) |
I'm going to add my experience here resolving similar (but perhaps different) issues. Problem Solution
$ docker network inspect bridge
[
{
"Name": "bridge",
"Id": "aec41fdfc6195074c49b330ce04a28e11d72788bc7204fe5f5ae2bf39b642a25",
"Scope": "local",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Config": [
{
"Subnet": "172.17.0.1/16",
"Gateway": "172.17.0.1"
}
]
},
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "9001"
}
}
]
Note: Docker check-config.sh is on github/docker/docker/contrib.
And now for some interactive tests (notice:
Note that without
Also, partial relevant contents of my
Just for kicks, let's make sure I'm really starting the Docker daemon on Ubuntu 14.04 using
Finally, since I'm on Ubuntu, let's do the thing people suggest often: Delete the
Okay, next post I will show what did resolve my lack of DNS access from within my containers. |
Okay, next day, same computer, same exact setup, but different network, and a new idea from this thread to try. I also recommend reading the latest Docker Networking guide fully: https://docs.docker.com/engine/userguide/networking/dockernetworks/ Note: I used explicit
Whoa! And then someone else here mentioned
I also noticed one thing here that I haven't isolated yet: Some documentation and discussion about these
And then took a deep breath:
Wow. I'm in business. And I can run I will possibly post one more here in this thread (or not), but I intend to re-verify from my home network if there is any difference or that it really was just my configuration changes ( I hope some of these two posts help others track down or try different ideas to investigate their problems. The above tips got me past my problem after banging my head for way too long on this strange matter. |
So, it really was an issue at home with my home WiFi router. Really unbelievable, but my D-Link DIR-655 has a built-in DNS server feature, which is described as "Advanced DNS is a free security option that provides Anti-Phishing to protect your Internet connection from fraud and navigation improvements such as auto-correction of common URL typos." I turned that garbage off and specified Google's 8.8.8.8 as the primary DNS server and Comcast (my home ISP's primary DNS server) as my secondary, rebooted it, and I can run the Docker daemon correctly at home. I've never had any kind of DNS issues with any other devices connected to this router until now; so I wonder what exactly the issue is with it not liking DNS requests coming from Docker?? Screenshot below. |
For what it's worth, the Advanced DNS feature of the D-Link DIR-655 is fully described as:
I'm wondering what Best Path Networks' particular problem could be. OpenDNS' engineering blog posts about Docker frequently, maybe when I have more time to dig into this I'll track what exactly in Best Path Network's filtering is causing false-positive filtering on my home router. |
I have the same DNS problem and I set |
I still have the same issue, but it doesn't have anything to do with the DNS. It's just the NATing rule in the POSTROUTING chain of NAT table which is missing. Once I restart docker daemon or add it manually, my containers can access internet again. But both of the solutions are just workarounds ! Restarting docker daemon will kill the running containers and adding the rule may cause other issues (overwriting other rules or just make them useless, etc ...). |
We have this similar problem too. Current theory is iptables rules getting overridden/removed by a puppet module running (via cron) on the hosts. Does that sound plausible for anyone else? |
I was getting this problem while building images and running containers on version 1.10.3, build 20f81dd and docker-machine version 0.6.0, build e27fb87. Both docker and docker-machine have their default configs. Restarting docker machine seems to be the only thing I tried that fixes it. Not sure if this will come back again as this is a fresh install of docker/docker-machine from Friday that worked fine then and didn't work at all until the restart today. |
We are having the same issue. After about 16-19 hours of uptime (longer on the weekends), the container goes into a state where it cannot talk to the outside world. Restarting the container or the docker daemon (which in turn restarts the container) will bring everything back to operating properly... for the next 16-19 hours or so. Inside the container is a node.js webapp. |
I found another possible problem, the privilege of container /etc/resolv.conf sometimes get changed to 600 weirdly, after docker restart or some special steps I am not sure, so for applications without root privilege will not be able to resolve the domain name, manually change it back to 644 will recover the issue. I encountered this issue in synology NAS DSM5.2 several times, finally figured out the problem... |
@poga , @bwinterton , @fbourigault : Did you guys find any solution for running arch linux systemd-networkd together with docker? I have been relying on this restarting docker service to fix the |
I'm having similar network issues as well, immediately after restarting docker I'm able to access the internet from the container but after about a dozen attempts it fails (see below). This is reproducible every time. However the network failure doesn't occur if I start a container in the success state and keep it running the whole time. 'ip route', 'iptables -L', and 'docker network inspect bridge' are identical in both the success and failed states. ... $ docker version Server: Log of container with network failure |
Same issue +1. I can't ping @aboch The output of
There's no rules there! And the docker daemon is run with Then I modified
I disable the iptables parameter again (changed to
|
Get same issue. Server: $ uname -a But in my case nothing helps, restart service, reboot system, reinstall docker, add new bridges nothing... I asked the question there http://askubuntu.com/questions/811895/ubuntu-16-04-iptables-on-postrouting-do-not-recognize-docker0-bridge |
Had the same trouble with building nginx:alpine What fixed it for me was that the build process uses docker0 and not the bridge. |
Just in case it helps folks running docker in an Ubuntu VM under Hyper-V (Windows) facing this same issue, it can be worked around by specifying the dns to use.
|
A clarification on your scenario, did you spin up an ubuntu VM then independently install docker on that machine, or are you using the VM that is automatically deployed and configured when you install the docker tools for windows? I'm doing the latter (automatic config) and I'm running into this network issue but when I try to connect to MobyLinuxVM through HyperV I'm unable to interact with the VM Update, reinstalling docker for windows fixed things... |
Hey @lukewaters! Glad to hear the issue went away for you. I'm talking about an Ubuntu VM I set up myself (in Hyper-V) and within this Ubuntu VM I'm running both the docker daemon and the docker client (Linux). |
I'm going to close and lock this issue, because this issue has become a collection of a wide range of DNS and network related issues. Having all these issues collected in a long thread makes it very difficult to look into (some reports were resolved, others were reported a long time ago, and may no longer be relevant). Also note that if you're having issues on Docker for Mac, please report through https://github.com/docker/for-mac/issues, as those issues can be specific to Docker for Mac (and not the "engine"). If you're still having this issue, please open a new issue with all the relevant information, so that it can be looked into in detail. Thanks in advance for doing so, and apologies for having re-submit your issue if you're still having this. |
When I ping google.com in the container, it return : ping: unknown host
[HOST Info]
root@host# uname -a
Linux localhost 4.0.2-x86_64-linode56 #1 SMP Mon May 11 16:55:19 EDT 2015 x86_64 GNU/Linux
root@host# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
start the container
root@host#docker run --rm -it debian /bin/bash
start capture package
root@host# tshark -i eth0 -i docker0
1 0.000000 106.186.. -> 8.8.8.8 DNS 70 Standard query 0xb49a A google.com
2 0.046688 8.8.8.8 -> 106.186.. DNS 86 Standard query response 0xb49a A 216.58.221.14
3 -0.000042 172.17.0.4 -> 8.8.8.8 DNS 70 Standard query 0xb49a A google.com
4 4.171017 fe80::1 -> ff02::1 ICMPv6 118 Router Advertisement from 00:05:73:a0:0f:ff
5 5.005167 106.186.. -> 8.8.8.8 DNS 70 Standard query 0xb49a A google.com
6 5.007502 8.8.8.8 -> 106.186.. DNS 86 Standard query response 0xb49a A 216.58.221.14
7 5.005127 172.17.0.4 -> 8.8.8.8 DNS 70 Standard query 0xb49a A google.com
8 5.016512 02:42:ac:11:00:04 -> ca:5b:7d:34:78:20 ARP 42 Who has 172.17.42.1? Tell 172.17.0.4
9 5.016542 ca:5b:7d:34:78:20 -> 02:42:ac:11:00:04 ARP 42 172.17.42.1 is at ca:5b:7d:34:78:20
10 10.010414 106.186.. -> 8.8.8.8 DNS 70 Standard query 0x1367 A google.com
11 10.046683 8.8.8.8 -> 106.186.. DNS 86 Standard query response 0x1367 A 216.58.221.14
12 10.010374 172.17.0.4 -> 8.8.8.8 DNS 70 Standard query 0x1367 A google.com
13 15.015578 106.186.. -> 8.8.8.8 DNS 70 Standard query 0x1367 A google.com
14 15.052782 8.8.8.8 -> 106.186.. DNS 246 Standard query response 0x1367 A 173.194.126.198 A 173.194.126.196 A 173.194.126.197 A 173.194.126.194 A 173.194.126.195 A 173.194.126.193 A 173.194.126.206 A 173.194.126.199 A 173.194.126.200 A 173.194.126.192 A 173.194.126.201
15 15.015538 172.17.0.4 -> 8.8.8.8 DNS 70 Standard query 0x1367 A google.com
root@f82d47432161:/# ip addr
eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.4/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe11:4/64 scope link
valid_lft forever preferred_lft forever
root@f82d47432161:/# ping google.com
ping: unknown host
It seems that the host did not forword package to the container
The text was updated successfully, but these errors were encountered: