Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
No networking in some docker containers when spawned at high rate #1638
When spawning docker container at high rate in network mode bridge, some containers will not have networking connectivity. Coming from moby/moby#27808 asked to report the bug in CoreOS
VM running on VMWare
Steps to reproduce the issue:
I can reproduce the issue with the following script, some container image work better but will fail at some point. Also, I can not reproduce the issue in host network mode.
Describe the results you received:
About 1% of the container will fail without networking.
Describe the results you expected:
No network failure, ping should go through, arp should resolve.
Additional information you deem important (e.g. issue happens only occasionally):
The issue only occur for 1% of the pods when the system is under load, spawning and deleting lots of containers.
I've run this a few times to ping the gateway with the default bridge network driver, and there have been no ping failures.
How did you set up your custom network? Can you confirm whether the failed containers are attached to the bridge? (For example,
It happens that the node I used for repro just reloaded and I couldn't reproduce the issue easily. I had to re-run several times the scripts (maybe 5000 container runs) until I found the first failures.
Nevertheless, here is
Here is one that worked:
Okay, I eventually managed to reproduce this with a fresh CoreOS system. However, when using an image built with a fix I've already proposed to systemd, I could not reproduce the issue (after nearly 20,000 containers spawned). We are waiting on upstream to decide on the configuration option to use in systemd/systemd#4228, but we can backport it to fix this issue when a decision is made.