Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upNo networking in some docker containers when spawned at high rate #1638
Comments
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
dm0-
Oct 31, 2016
Member
I've run this a few times to ping the gateway with the default bridge network driver, and there have been no ping failures.
for ((i=0; i<200; i++)) ; do (docker run --rm busybox ping -c 1 172.17.0.1 &> fail.$i && rm -f fail.$i) & done
How did you set up your custom network? Can you confirm whether the failed containers are attached to the bridge? (For example, ip link should show e.g. master docker0 on the veth interface.)
|
I've run this a few times to ping the gateway with the default bridge network driver, and there have been no ping failures. How did you set up your custom network? Can you confirm whether the failed containers are attached to the bridge? (For example, |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rawouter
Oct 31, 2016
It happens that the node I used for repro just reloaded and I couldn't reproduce the issue easily. I had to re-run several times the scripts (maybe 5000 container runs) until I found the first failures.
Nevertheless, here is ip link from a container that failed:
PING 173.36.21.105 (173.36.21.105) 56(84) bytes of data.
From 173.77.4.23 icmp_seq=1 Destination Host Unreachable
--- 173.36.21.105 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
8274: eth0@if8275: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:ad:4d:04:17 brd ff:ff:ff:ff:ff:ff
Here is one that worked:
PING 173.36.21.105 (173.36.21.105) 56(84) bytes of data.
64 bytes from 173.36.21.105: icmp_seq=1 ttl=61 time=3.71 ms
--- 173.36.21.105 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.716/3.716/3.716/0.000 ms
8192: eth0@if8193: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:ad:4d:03:ed brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
rawouter
commented
Oct 31, 2016
•
edited
Edited 1 time
-
rawouter
edited Nov 1, 2016 (most recent)
edited
-
Nov 1, 2016 (most recent)
|
It happens that the node I used for repro just reloaded and I couldn't reproduce the issue easily. I had to re-run several times the scripts (maybe 5000 container runs) until I found the first failures. Nevertheless, here is Here is one that worked: |
crawford
added
kind/bug
component/docker
priority/P1
team/os
area/stability
labels
Nov 1, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
dm0-
Nov 2, 2016
Member
Okay, I eventually managed to reproduce this with a fresh CoreOS system. However, when using an image built with a fix I've already proposed to systemd, I could not reproduce the issue (after nearly 20,000 containers spawned). We are waiting on upstream to decide on the configuration option to use in systemd/systemd#4228, but we can backport it to fix this issue when a decision is made.
|
Okay, I eventually managed to reproduce this with a fresh CoreOS system. However, when using an image built with a fix I've already proposed to systemd, I could not reproduce the issue (after nearly 20,000 containers spawned). We are waiting on upstream to decide on the configuration option to use in systemd/systemd#4228, but we can backport it to fix this issue when a decision is made. |
crawford
added this to the CoreOS Alpha 1263.0.0 milestone
Dec 2, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
dm0-
Dec 3, 2016
Member
The proposed option was merged in upstream systemd, and we are going to backport it to our current systemd versions at coreos/systemd#73.
|
The proposed option was merged in upstream systemd, and we are going to backport it to our current systemd versions at coreos/systemd#73. |
rawouter commentedOct 31, 2016
Issue Report
Bug
When spawning docker container at high rate in network mode bridge, some containers will not have networking connectivity. Coming from moby/moby#27808 asked to report the bug in CoreOS
CoreOS Version
Environment
VM running on VMWare
Steps to reproduce the issue:
I can reproduce the issue with the following script, some container image work better but will fail at some point. Also, I can not reproduce the issue in host network mode.
Describe the results you received:
About 1% of the container will fail without networking.
Here some command outputs taken from the script above:
Describe the results you expected:
No network failure, ping should go through, arp should resolve.
Additional information you deem important (e.g. issue happens only occasionally):
The issue only occur for 1% of the pods when the system is under load, spawning and deleting lots of containers.
It doesn't occur in host network mode.
Output of
docker version:Output of
docker info: