Timeout: UnixHTTPConnectionPool

[ureyes84]

We are spawning multiple containers from multiple processes concurrently. It's hard to tell exactly when this happens but we have noticed this is more frequent the more containers we start.

Docker version

Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f
OS/Arch (client): linux/amd64
Server version: 1.3.0
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): c78088f

Docker client

docker-py==0.6.0
version='1.14', base_url='unix://var/run/docker.sock'

Here is the exception we are getting:

  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/docker/client.py", line 974, in start
    res = self._post_json(url, data=start_config)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/docker/client.py", line 237, in _post_json
    return self._post(url, data=json.dumps(data2), **kwargs)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/docker/client.py", line 73, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/requests/sessions.py", line 425, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/requests/adapters.py", line 387, in send
    raise Timeout(e)
Timeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

What does this mean? Timeout seems high already, would a retry help?

[ureyes84]

We are also seeing the following error during the same conditions as above. It almost seems like docker is getting overwhelmed:

File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/docker/client.py", line 975, in start
    self._raise_for_status(res)
  File "/home/mark/.virtualenvs/dev/local/lib/python2.7/site-packages/docker/client.py", line 89, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error ("Cannot start container 6b6e90804cd041fb4bbb5b7872bf2c4ec8b8086a4054a87cd76ec35826d2267a: Process could not be started")

[shin-]

Sounds to me like you're overloading your machine. What are the specs? How many containers are you running concurrently?

[ureyes84]

Seems plausible. We are lunching 14 containers with heavy processing on 1 computer with 32 cores and 128 GB of RAM

[mr337]

@ureyes84
When this happens are you having issues killing the container. I have encountered this and once this happens leaves Docker in a really slow state where Docker commands either hang or super slow.

Example

ubuntu@ip-10-171-68-239:/var/log/upstart$ time docker rm e26a081ce5ec
e26a081ce5ec

real    0m44.090s
user    0m0.020s
sys     0m0.001s

and that was with everything idling.

[weiwongfaye]

it happens to me today, there are 50 container start, and then the message appear. system became very slow.

[mr337]

Finally found the root issue that was messing me up. We are running Docker on EC2 instances, without realizing it we was taxing the EBS volume by requesting too many writes and reads and it couldn't burst anymore without getting a bigger EBS volume. So in the end boiled back to overloading the host machine.

[shin-]

Closing, evidence points to this not being a docker-py issue.

[vitalyisaev2]

I face this issue regularly because we use docker-py for launching of dozens (and even hundreds) containers every day from our Jenkins. This is what I usually see:

  File "/usr/local/bin/package-create", line 9, in <module>
    load_entry_point('docker-buildscripts==0.2.39', 'console_scripts', 'package-create')()
  File "/usr/local/lib/python2.7/dist-packages/buildscripts/create.py", line 154, in main
    manager, runtime_config, aptly, proj, kwargs = build_params(options)
  File "/usr/local/lib/python2.7/dist-packages/buildscripts/create.py", line 125, in build_params
    assert(all(map(manager.image_exists, runtime_config['images'].values())))
  File "/usr/local/lib/python2.7/dist-packages/buildscripts/proxy/_docker.py", line 22, in image_exists
    return self.images(name=image_name)
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 640, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 106, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 433, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

Docker==1.6.0
docker-py==1.2.3

[dnephin]

If you get a timeout on the client it's almost always an issue with the server. 60s should be more than enough time to wait for a response. Since you're running such an old version of docker, it's probably worth upgrading.