container failed to start due to network issue #14788
Labels
Comments
|
@quick-sort can you post your daemon options? most likely you started the daemon with an existing bridge which has a smaller subnet range or you used |
|
@mrjana I didn't use --fixed-cidr option, I used the default docker network. But I did remove and create containers again and again for hundreds of times, and it worked fine after I have restarted docker daemon. |
|
@quick-sort do I understand correctly that you're no longer able to reproduce this (after the daemon was restarted)? |
|
@theJeztah yes, after the daemon was restarted, I cannot reproduce it. |
|
@quick-sort Thanks! I'll close this issue for now, but please comment here / ping me, if you're able to reproduce and have more info 👍 |
|
Same here. Disappeared after daemon restart. |
|
|
@thaJeztah, I can't reproduce this on demand, but we see this state routinely. Restarting the Docker daemon does work, but causes downtime. What information can I get for you next time we see it? |
|
Yep, 1.8.2. |
|
@thaJeztah @mrjana ping. I'd love to help get this resolved. |
|
Ok, let me reopen this ping @mrjana #14788 (comment) |
|
@apatil what is your bridge configuration? i.e what kind of subnet are you using? |
|
@mrjana it's a Flannel bridge with a /24 subnet. There were no IP address exhaustion issues with the same setup on Docker 1.6. The problem is occurring now on one of our machines. There are 29 containers running, and 14 of them share the network namespace of another container, so there are plenty of IP addresses available in the subnet. |
|
+1 happening to me both on Docker 1.7.1 and 1.8.3. ~25 containers running on a /24 with flannel on 835.8.0. Had 80 days uptime until a few days ago when etcd started acting up and all of the sudden this started happening. Rebooted, no improvement. Upgraded, no improvement. |
|
Any updates on this? |
|
This is happening in my kubernetes cluster (docker 1.7.1). Any news? |
|
Here's a systematic repro (from kubernetes/kubernetes#19477 (comment) ): Which apparently doesn't happen with Docker 1.9 |
This was referenced Jan 14, 2016
|
ping @mrjana can you have another look at this if the new information mentioned above is useful for reproducing? |
|
@2opremio Is it possible for you to upgrade to 1.9 (or atleast to 1.8)? We fixed a fundamental namespace related issue in 1.8 which can sometimes show up in different manifestations (such as yours) and so I would suggest upgrading to newer release. |
|
@mrjana The problem is reproducible with 1.8 as stated above, so upgrading to 1.8 won't help. Also, we cannot upgrade to 1.9 due to #17720, which is a blocker for kubernetes due to its aggresive polling of |
|
@2opremio Did you test master or 1.10-rc1 to see if #17720 has been resolved (or atleast there is no appreciable slowness)? We have completely rewritten ip allocator code in 1.9 so it won't be worthwhile to go fix some code (the ip allocator in 1.7 and 1.8) which is now obsolete. If you don't see #17720 in 1.10-rc1 then upgrading to 1.10 might be the option since it is just around the corner. |
|
Given the above comments state this specific issue was fixed in 1.9.0 and we had not heard of this specific problem being reproduced on the past two docker versions (1.12.1 and 1.11.2) I am suggesting to close this issue. |
|
Closing this due to comments. Feel free to open new issue if the problem still persists. |
|
I'm seeing this on 1.11.2 and 1.13.0 root@myhost1# docker info root@myhost2# docker info It happens frequently during a crash situation, but is happening at background levels unassociated with noticeable degradation at a rate of 6 events over a 5 minute period every few days. Then when there is a serious event, like Our environment is vmware running rancher. We also see it in aws running rancher. |
|
@kbroughton Did you find a fix for this with Rancher? |
|
@kbroughton any successful workaround so far ? |
|
I encounter the same issue on vmware which cause |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
docker version
docker info
uname -a
I am not sure how to reproduce this issue, because it doesn't occur on another docker host. I used docker-py to do operation like run/rm container.
I could start new container as long as the number of running containers is less than 5. However I cannot start any new container when there are five containers running.
docker run -d --name nginx nginx
It will give "Cannot start container xxxxx: no available ip addresses on network"
After service docker restart, it can start new containers successfully.
I found this in syslog
It seems that new veth failed to start properly caused this issue. What else should I check in this docker host?
----------END REPORT ---------
The text was updated successfully, but these errors were encountered: