No networking in some docker containers when spawned at high rate #1638
Comments
I've run this a few times to ping the gateway with the default bridge network driver, and there have been no ping failures.
How did you set up your custom network? Can you confirm whether the failed containers are attached to the bridge? (For example, |
It happens that the node I used for repro just reloaded and I couldn't reproduce the issue easily. I had to re-run several times the scripts (maybe 5000 container runs) until I found the first failures. Nevertheless, here is
Here is one that worked:
|
Okay, I eventually managed to reproduce this with a fresh CoreOS system. However, when using an image built with a fix I've already proposed to systemd, I could not reproduce the issue (after nearly 20,000 containers spawned). We are waiting on upstream to decide on the configuration option to use in systemd/systemd#4228, but we can backport it to fix this issue when a decision is made. |
The proposed option was merged in upstream systemd, and we are going to backport it to our current systemd versions at coreos/systemd#73. |
Issue Report
Bug
When spawning docker container at high rate in network mode bridge, some containers will not have networking connectivity. Coming from moby/moby#27808 asked to report the bug in CoreOS
CoreOS Version
Environment
VM running on VMWare
Steps to reproduce the issue:
I can reproduce the issue with the following script, some container image work better but will fail at some point. Also, I can not reproduce the issue in host network mode.
Describe the results you received:
About 1% of the container will fail without networking.
Here some command outputs taken from the script above:
Describe the results you expected:
No network failure, ping should go through, arp should resolve.
Additional information you deem important (e.g. issue happens only occasionally):
The issue only occur for 1% of the pods when the system is under load, spawning and deleting lots of containers.
It doesn't occur in host network mode.
Output of
docker version
:Output of
docker info
:The text was updated successfully, but these errors were encountered: