exception ReadTimeout: UnixHTTPConnectionPool when calling container.stop()

[pommetjehorlepiep]

OS: Debian 8 Jessie 64-bit

pip show docker:
Name: docker
Version: 2.0.0
Summary: A Python library for the Docker Engine API.
Home-page: https://github.com/docker/docker-py
Author: Joffrey F
Author-email: joffrey@docker.com
License: UNKNOWN
Location: /usr/local/lib/python2.7/dist-packages
Requires: backports.ssl-match-hostname, ipaddress, docker-pycreds, websocket-client, requests, six

docker version:
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:21:54 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:21:54 2016
OS/Arch: linux/amd64

ReadTimeout exception:

File "/usr/local/lib/python2.7/dist-packages/docker/models/containers.py", line 343, in stop
return self.client.api.stop(self.id, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 21, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/api/container.py", line 1084, in stop
timeout=(timeout + (self.timeout or 0)))
File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 47, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 171, in _post
return self.post(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 535, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 499, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)

[pommetjehorlepiep]

Hack to get around the problem:

catch requests.exceptions.ReadTimeout exception and call container.kill()

If you don't call kill() container still seen as running

[TomasTomecek]

Can you please post a minimal reproducer which we would run on our laptops? Also please post what is the content of the container.

[pommetjehorlepiep]

Broad steps on how to reproduce

1. Initial image: Debian 8 Jessie. Used debootstrap to create an image, imported the tar file into docker
2. Using apt-get install apt-cacher-ng in a container using the initial image
3. Commit the container with apt-cacher-ng as new image (repo:tag: apt-cacher-ng:base)
4. Create a container using image apt-cacher-ng:base

client = docker,from_env()

client.containers.create(
image='apt-cacher-ng:base'
,command='/usr/local/bin/runner'
,detach=True
,environment=['ADMIN_PWD=password']
,hostname='apt-cacher-ng'
,labels=['apt-cacher-ng']
,name='apt-cacher-ng'
,network_mode='bridge'
,ports={'3142/tcp' : 3142 }
,tty=False
,user='apt-cacher-ng:apt-cacher-ng'
,volumes={ '/docker/images/apt-cacher-ng/containers/apt-cacher-ng' : { 'bind': '/mnt/scripts', 'mode': 'rw' }}
)

/usr/local/bin/runner: A shell script which loads some library functions and then calls a script in /mnt/scripts which runs apt-cacher-ng in the foreground using exec

5. Once container created, start container using docker.from_env().containers.container.start()
6. Check container up and running
7. Now try to stop the running container
   1. Find the running container using containers.find('apt-cacher-ng')
   2. Once found, do ..containers.container.stop()
      Exception is thrown after 10 seconds

[TomasTomecek]

Can you also reproduce using docker client?

[pommetjehorlepiep]

Using docker stop apt-cacher-ng works fine. No problems here.

[cjh1]

I am seeing this as well

[AnthonyDeroche]

Hello, I'm experiencing the same issue with docker-py 2.3.0.
Have you planned to have a look at this soon? I dont know if the docker cli is using the REST API as well, but I do not encounter the problem with the cli.
Regards,
Anthony

[grahamlyons]

I also see this problem with the latest docker-py.

Minimal steps to reproduce, on CentOS 7 with docker-ce-17.03.1:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro python:3 \
pip install docker && \
python -c "import docker; docker.from_env().containers.run('alpine', detach=True, command=['/bin/sh', '-c', 'sleep 20']).stop()"

Results in:

requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=10)

It looks like the timeout value for the stop method might be being passed to requests.

Update:
The logic in the stop method on the API takes the stop timeout and passes it as a parameter to the Docker API, then it sets the HTTP read timeout to that plus whatever timeout is set for the client, defaulting to 0:
https://github.com/docker/docker-py/blob/dc2b24/docker/api/container.py#L1117

The initialisation of the client defaults that timeout to DEFAULT_TIMEOUT_SECONDS, which is 60:
https://github.com/docker/docker-py/blob/dc2b24/docker/api/client.py#L87

However, using the from_env method of the DockerClient class sets the timeout argument to None:
https://github.com/docker/docker-py/blob/dc2b24/docker/client.py#L76

This means that the request times out at the same point that the container is forcefully stopped so we don't see the request return and the exception is raised from requests.

[shin-]

Should be fixed by #1634

[orodbhen]

I'm still seeing this in 3.4.1:

OS: CentOS 7.5
Docker: 18.03

As before, it does not occur with the docker CLI, only the Python API.

In my case it occurs when running container.wait(). It originates with a socket.timeout exception in the urllib3 package, which sets off a cascade of other exceptions:

• socket.timeout
• urllib3.exceptions.ReadTimeoutError
• requests.exceptions.ConnectionError: UnixHTTPConnectionPool

It seems that requests is raising the wrong exception: requests.exceptions.ConnectionError instead of requests.exceptions.ReadTimeout. Shouldn't these exceptions be handled in the Docker API and converted to a public exception, rather than relying on the exception type from an underlying package?