UnixHTTPConnectionPool(host='localhost', port=None): Pool is closed

[ReeSilva]

ISSUE TYPE

• Bug Report

SUMMARY

When we have a high number of containers, Docker Compose exceed the limit of connection pool. This was fix on recently released docker-compose, but Ansible Container still uses Docker Compose.

If possible this could be changed on the actual version of Ansible Container, once this can block of use who have a lot of containers.

ERROR: for folder-queue-listener  UnixHTTPConnectionPool(host='localhost', port=None): Pool is closed.
UnixHTTPConnectionPool(host='localhost', port=None): Pool is closed.

STEPS TO REPRODUCE

Create an structure with more than 20 containers.
Try to run Ansible-container.

References

docker/compose#3586
https://github.com/docker/compose/releases/tag/1.8.1

[chouseknecht]

Pushed a new ansible/ansible-container-builder:0.2 to Docker Hub just a few minutes ago. Here's what is actually installed:

# yum list install "docker*"
Loaded plugins: fastestmirror, ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered with Subscription Management. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirror.cs.pitt.edu
 * epel: ftp.linux.ncsu.edu
 * extras: mirror.cs.pitt.edu
 * updates: mirrors.advancedhosters.com
Installed Packages
docker-common.x86_64                                                                           1.10.3-46.el7.centos.14                                                            @extras
docker-latest.x86_64                                                                           1.12.1-2.el7.centos                                                                @extras
docker-selinux.x86_64                                                                          1.10.3-46.el7.centos.14                                                            @extras
Available Packages
docker.x86_64                                                                                  1.10.3-46.el7.centos.14                                                            extras
docker-devel.x86_64                                                                            1.3.2-4.el7.centos                                                                 extras
docker-distribution.x86_64                                                                     2.4.1-2.el7                                                                        extras
docker-forward-journald.x86_64                                                                 1.10.3-44.el7.centos                                                               extras
docker-latest-logrotate.x86_64                                                                 1.12.1-2.el7.centos                                                                extras
docker-latest-v1.10-migrator.x86_64                                                            1.12.1-2.el7.centos                                                                extras
docker-logrotate.x86_64                                                                        1.10.3-46.el7.centos.14                                                            extras
docker-lvm-plugin.x86_64                                                                       1.10.3-46.el7.centos.14                                                            extras
docker-novolume-plugin.x86_64                                                                  1.10.3-46.el7.centos.14                                                            extras
docker-python.x86_64                                                                           1.4.0-115.el7                                                                      extras
docker-registry.noarch                                                                         0.6.8-8.el7                                                                        extras
docker-registry.x86_64                                                                         0.9.1-7.el7                                                                        extras
docker-unit-test.x86_64                                                                        1.10.3-46.el7.centos.14                                                            extras
docker-v1.10-migrator.x86_64                                                                   1.10.3-46.el7.centos.14                                                            extras

@ReeSilva can you confirm if this fixes the issue you're seeing?

[ReeSilva]

@chouseknecht I will try it on my next builds and see if error occur again. Does ansible-container is already using ansible-container-builder:0.2? I'm using ansible-container from pip version, once we use this to build our staging containers.

[chouseknecht]

To use the 0.2 container, you will need to install from source, not from pip.

[ReeSilva]

@chouseknecht Ok. On the 0.2 Ansible Container will change anything about how I call my playbooks or I prepare my environment to run Ansible-Container? Because if I have to rewrite this I belive that u guys still don't have docs to explain, and I will lost sometime to find out how to do.

But, if there is no change in this stuffs and I can run Ansible Container with my files the way they are now, I'll give a try. In other way, if I have to change something I'll wait for the release on pip.

[chouseknecht]

You should not need to change anything. In version 0.2 we have been fixing bugs and adding new features. I can't think of anything that got removed or redefined.

Of course, if this is a production process, I would recommend testing, just to make sure I'm not forgetting something.

[ReeSilva]

@chouseknecht Nice to know. So tomorrow I'll give a try and test. If everything works well, I'll implement. Thanks a lot.

[ReeSilva]

Still getting the error even with the new version. This is an error with docker-compose, that was fixed on 1.8.1 (docker/compose#3586)

Ansible Container, version 0.2.0-pre
Linux, ip-172-31-18-38, 3.13.0-74-generic, #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015, x86_64
2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] /usr/bin/python

[chouseknecht]

Hmmm... we're installing docker in the ansible/ansible-container-builder:0.2 image, because we need the client pieces. We install the latest available RPM which is 1.12.1. It looks like that was released around August 18, so it would not have this change.

The next release after that is 1.12.2-rc1. Not sure where to get the RPM for that, but it's probably available. What you could is...

• clone the Ansible Container repo
• modify container/templates/ansible-dockerfile.j2 line 8 to install the new RPM. (You may also have to first add the Docker RPM repo.)
• remove any builder images i.e. docker rmi --force ansible/ansible-container-builder:0.2 and docker rmi --force ansible-container-builder
• and finally, create a local build image via: ansible-container build --local-builder

[ReeSilva]

Does docker-compose comes with docker-latest?

[chouseknecht]

You get the docker-compose CLI tool. It doesn't install the python module, but presumably whatever fixes are baked into the python somehow end up in the CLI. That's my assumption.

[j00bar]

Docker Compose is run from outside of the build container in Ansible Container itself. The requirement is defined here: https://github.com/ansible/ansible-container/blob/develop/requirements.txt#L1

[chouseknecht]

Ugh. Totally missed that.

Is there any reason to not bump this to 1.8.1 and just let it install the latest docker-py?

[rinatz]

I have same problem on commiting large size docker container.
But when passed a timeout value to docker.AutoVersionClient, this problem didn't cause.
rinatz@66017cf

[TomasTomecek]

It would be really great if we could change timeout via environment variable (likely done in here) b/c right now it's hardcoded.

[rinatz]

@TomasTomecek
When docker-compose creates docker.Client instance, it does like that.
https://github.com/docker/compose/blob/master/compose/cli/docker_client.py#L44

[rinatz]

Sorry, this problem has no relation to my comment.

#261: UnixHTTPConnectionPool(host='localhost', port=None): Pool is closed.
#147: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

[chouseknecht]

Working on upgrading to latest docker-py and docker-compose libraries: #314. PR in place here: #316

Thinking we really need docker/compose#4090 resolved before we merge, even though all tests are passing.

[chouseknecht]

If you need to set a longer timeout for the docker client, the develop branch now respects DOCKER_CLIENT_TIMEOUT and COMPOSE_HTTP_TIMEOUT.

[levonet]

I had the same error when starting docker with log-driver: syslog when rsyslog port was unavailable.
Error starting container 0ba2fb9540ec6680001f90dce56ae3a04b831c8146357efaab79d4756253ec8b: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)