Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot connect from Docker containers to the outside #1936

pietervisser opened this issue Apr 26, 2017 · 8 comments

Cannot connect from Docker containers to the outside #1936

pietervisser opened this issue Apr 26, 2017 · 8 comments


Copy link

@pietervisser pietervisser commented Apr 26, 2017

Issue Report


Since the update to 1353.6.0 we experience network issues when trying to ping from a docker container to the outside.

Container Linux Version

NAME="Container Linux by CoreOS"
PRETTY_NAME="Container Linux by CoreOS 1353.6.0 (Ladybug)"


Google Cloud

Expected Behavior

Pinging to from a Docker container in a docker network should result in no pakket loss.

Actual Behavior

When creating a new docker network and trying to connect to the world, sometimes a network is created in which no connections are possible to the outside.

  1. Create a docker network
  2. Run a container and ping to the outside
  3. Remove the network

Repeat this several times, and sometimes 100% packet loss.

docker network create foo > /dev/null; docker run --rm --net foo busybox ping -c 1 -w 1 -q; docker network rm foo > /dev/null

@pietervisser pietervisser changed the title Random network issues with Docker network Cannot connect from Docker containers to the outside Apr 26, 2017
Copy link

@lucab lucab commented Apr 26, 2017

@pietervisser I guess you are running the above command in a while loop. Can you please check if you can reproduce the same issue with unique network names (e.g. foo$i with a monotical index, instead of foo)? I know there are some potential races in network creation and I'm not sure if you are hitting that or something else.

Copy link

@pietervisser pietervisser commented Apr 26, 2017

@lucab, thanks but no I'm not running it in a while loop. Executing this just a couple of times manually will result in connection issues. To be sure, I used unique networks and can still reproduce this issue. Use could use this loop to reproduce the issue.

for i in {1..10}; do docker network create foo$i > /dev/null; docker run --rm --net foo$i busybox ping -c 1 -w 1 -q; docker network rm foo$i > /dev/null; done

Copy link

@bsphere bsphere commented Apr 26, 2017

we also experience the same thing on AWS.

re-running ping multiple times (inside a user create network container) sometimes work and sometimes not. i don't have to re-create the network..

this update screwed up an ~40 nodes nomadproject cluster for us.

Copy link

@dm0- dm0- commented Apr 26, 2017

Can you try this and see if it makes a difference? Copy /usr/lib/systemd/network/ into /etc/systemd/network, and change the Match=docker* line to Match=docker* br-*. Then run sudo systemctl restart systemd-networkd. Does that fix the issues?

Copy link

@bsphere bsphere commented Apr 26, 2017

@dm0- seems like it does make a difference, but I had to reboot for this to work.

Copy link

@dm0- dm0- commented Apr 26, 2017

We'll build a new stable with that fix which should be released over the next day.

Copy link

@euank euank commented Apr 26, 2017

To clarify the actual issue: this is basically a redux of #1554, but for the bridge interfaces docker network creates rather than the default docker0 interface.

I'm not sure the exact root cause. I can't reproduce this on the old stable and the docker version there was identical.
It seems like sometihng about networkd in systemd v233 changed which caused it to break these bridges, but I'm not totally sure!

We'll add a test for docker network create bridges to make sure this doesn't regress again; for the previous issue we were only testing it was fixed on docker0 and so missed this.

Thanks for reporting.

Copy link

@polygox polygox commented Apr 28, 2017

I am experiencing network problems with the current version 1353.7.0 which did not occur before (at least not before 1353.6.0, I am not sure when this happened first).

I use docker-compose for starting some apps and a proxy.
Two bridge networks are defined in the compose file.

When the coreos server is restarted (or the networks are removed), the apps are accessible from the outside when using docker-compose up (networks are created in this case). But when stopping all app/proxy-containers and starting them again, they are not accessible any more. Could this be related to this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants
You can’t perform that action at this time.