Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) #3927

Closed
bodaso opened this issue Sep 9, 2016 · 163 comments

Comments

@bodaso
Copy link

bodaso commented Sep 9, 2016

Hi since yesterday I've been running into this error while doing docker-compose up

Full Error Message

Device-Tracker $ docker-compose up
Creating device-tracker-db
Creating device-tracker

ERROR: for web  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "contextlib.py", line 35, in __exit__
  File "compose/cli/errors.py", line 56, in handle_connection_errors
TypeError: log_timeout_error() takes exactly 1 argument (0 given)
docker-compose returned -1

Docker Version
Docker for Mac: 1.12.0-a (Build 11213)
Machine info
MacBook Air (13-inch, Early 2015)
Processor: 1.6 GHz i5
Memory: 4GB 1600 MHz DDR3
macOS: Version 10.11.6 (Build 15G1004)

Attempts

  • Everything still works on colleagues' machine, they are using MacBook Pro
  • Increased Docker CPU from 2 to 3, and 2GB RAM to 3GB, still error
  • Removed All Docker containers & images, and rebuild everything, still error
@bodaso
Copy link
Author

bodaso commented Sep 9, 2016

tried this

export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120

and it seems to fix the issue for now

Other solutions people mentioned in this thread:

  • Restart Docker
  • Increase Docker CPU & memory

@shin-
Copy link

shin- commented Sep 9, 2016

Does it happen if you turn off your WiFi? Could be related to docker/docker-py#1076.

Another theory, if your service has tty: True enabled, could be #3106

@gvilarino
Copy link

I'm seeing exactly the same problem with latest beta for Mac. Same error if I run docker-compose create

Could this be related to having one very large layer in the image? (a very lengthy npm install operation that takes about a minute to be flattened into a layer when docker builds the image)

@benferns
Copy link

We are also seeing this issue using a docker compose file with 6 containers [docker-compose version 1.8.1, build 878cff1] on both windows and mac [Version 1.12.2-rc1-beta27 (build: 12496)
179c18cae7]

Increasing resources available to docker seems to reduce the chance of it happening (as does extending the timeout vars) , but its never eliminated.

We also have some large-ish layers (240MB is the largest, the main package install command) and we are binding to a host directory with 120MB of files across a couple of containers.

@gvilarino
Copy link

gvilarino commented Oct 11, 2016

From different attempts at working around this, I found something that might shed some light on a possible fix:

At first my scenario looked a bit like this:

app:
  build: .
  volumes:
    - ${PWD}:/usr/src
    - /usr/src/node_modules

My mounted path included many directories with big, static files that I didn't really need mounted in terms of code reloading. So i ended up swapping for something like this:

app:
  build: .
  volumes:
    - ${PWD}:/usr/src
    - /usr/src/static  # large files in a long dir structure
    - /usr/src/node_modules

This left out of the runtime mounting all my big static files, which made the service start way faster.

What I understand from this is: the more files you mount, especially the larger they are (images in the MBs instead of source files in the Bs/KBs), loading times go up by a lot.

Hope this helps

@WayneYe
Copy link

WayneYe commented Jun 12, 2017

+1
I am seeing this timeout issue every single week, usually after an idle weekend, while I was trying to connect to me containers, it timed out...
I have to terminate the running docker proc and restart it to work around....

@antoniolibrada
Copy link

+1
It happens to me every time I try to restart the containers because they are not responding anymore after a day. I'm not sure if my case has to do with the mounting since I am trying to stop the containers.

@luckydonald
Copy link

luckydonald commented Jun 14, 2017

Happinging with a nginx conatiner, Up 47 hours.
Docker for mac Version 17.03.1-ce-mac12 (17661) Channel: stable d1db12684b.

version: '2.1'
services:
  nginx:
    hostname: web
    extends:
      file: docker/docker-compose.yml
      service: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./src:/var/www:ro

  php:
    build:
      dockerfile: "./docker/web/php/Dockerfile"
      context: "."
    volumes:
      - ./src:/var/www
$ docker-compose kill nginx
Killing project_nginx_1 ... 

ERROR: for project_nginx_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

@cherrot
Copy link

cherrot commented Aug 8, 2017

Thanks @gvilarino, I believe the big files mounting is the cause of this issue on my linux server. Your snippet could be a workaround if the big files are not needed in container.

However, I wonder why mounting is slow in Docker? Maybe it triggers disk copy? But why?

@gvilarino
Copy link

@cherrot I wouldn't say I'm extremely proficient in the subject, but I believe this has to do with the storage driver used by Docker and how it works internally for keeping layers in order. Use docker info to see what storage driver your daemon is using (probably aufs, which is the slowest) and depending on your host OS, you may change it so something else (overlay being a better choice, if supported). There are faster alternatives like LCFS but they aren't commercially supported by Docker so you'd be on your own there.

@jcberthon
Copy link

We are also seeing this time-out. It seems also due to the volumes we are using.

We need some containers to access some SMB network shares. So we mounted those share on the host system, and bind-mounted them inside the container. But sometimes the communication between the Windows Server and our Linux host is stalled (see https://access.redhat.com/solutions/1360683) and this is blocking the starting or stopping of our container which just time-out after awhile.

I do not have a fix yet. I'm looking for a volume plugin which support SMB, or to make the stall communication problem on SMB going away. but no real solution yet.

@nvaken
Copy link

nvaken commented Apr 12, 2018

FWIW: For the people landing here through search engine finding their resolvement, I've been able to fix this simply by the did you try turning it off and on again? method; I've restarted my Docker Mac OS client.

@Binternet
Copy link

+1 on that, I am running stress testing on my instance which runs 4 containers and docker hangs even for docker ps -a so i'm trying to restart the containers but i am getting
UnixHTTPConnectionPool(host='localhost', port=None): Read timed out and

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 9, in <module>
    load_entry_point('docker-compose==1.8.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 61, in main
    command()
  File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 113, in perform_command
    handler(command, command_options)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/compose/cli/errors.py", line 56, in handle_connection_errors
    log_timeout_error()
TypeError: log_timeout_error() takes exactly 1 argument (0 given)

Only if im restarting the docker service it seems to be resolved, any ideas?

@Hyvi
Copy link

Hyvi commented May 4, 2018

+1

`Restarting web-jenkins_jenkins_1 ...

ERROR: for web-jenkins_jenkins_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=130)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 120).`

i restart docker, it solved. but every day i need to restart

@rodrigo-brito
Copy link

restarting Docker works to me.

@jamesdbruner
Copy link

+1 restarting docker worked for me as well.

@rafaelmagu
Copy link

I encountered this issue while building a substantially large Docker image and then attempting to push it to a remote registry. Restarting Docker wasn't an applicable solution, but @BODAZ' answer addressed it for me: #3927 (comment)

@SalathielGenese
Copy link

SalathielGenese commented Jul 4, 2018

@rodrigo-brito - I've been getting this error for a little while now and restarting docker deamon have been solving the issue - no more since I added another service to my project.

@bjesuiter
Copy link

I have the same problem, but I have a fairly simple setup.
I've only one verdaccio 3 container based on an image with 164 MB in size.
This is very disappointing :/

I'm using a MBP Pro 13 from 2015

@ericiper
Copy link

Happened to me because of a large port range, it actually creates one rule per port....

@ofekp
Copy link

ofekp commented Oct 6, 2018

A simple sudo service docker restart solves this for me consistently every time it occurs.

@trajano
Copy link

trajano commented Nov 20, 2018

Just happened to me as well, in my case docker-compose push (not even trying to run the app) on Azure DevOps.

My other builds do not use docker-compose but plain docker push

@AAber
Copy link

AAber commented Nov 21, 2018

I removed the kubuntu 18.04.1 docker.io version of docker and installed docker-ce 18.09.0
Problem went away.

@trajano
Copy link

trajano commented Nov 21, 2018

I just converted the docker-compose push step into individual pushes instead.

@ns-cweber
Copy link

ns-cweber commented Nov 29, 2018

We're seeing this timeout when running a container via docker-compose or via the docker-py library (times out even after we bump the timeout to 2 minutes); however, we don't see the error when we run via the Docker CLI (container starts instantly). We also only see the issue on a Linux CI server and not on our Macs. We're working on building out a minimal reproducible example.

@hazcod
Copy link

hazcod commented Dec 31, 2018

Having this issue with a docker-compose kill on a debian VM on macos host, install straight from docker. (Docker version 18.09.0, build 4d60db4)

@lacriment
Copy link

Having this issue "sometimes" on EC2 instance when trying to deploy with Gitlab Runner. Any help?

@Tempest82
Copy link

Tempest82 commented Jul 6, 2022

I had this issue as well and it was actually due to logging (in my case AWS CloudWatch) not being able to start up and connect to the proper servers due to a network security issue.

First I extended my timeouts to 3 minutes (two just wasn't enough)

export DOCKER_CLIENT_TIMEOUT=180
export COMPOSE_HTTP_TIMEOUT=180

Then I got the
failed to initialize logging driver: failed to create Cloudwatch log stream: RequestError: send request failed caused by: {{AWS URL and IP here}}: i/o timeout
Resolving the real issue resulted in everything working again. I hope in the future if the logging fails to start up it can be a better error than standard i/o timeout. You can add the longer timeouts to your service file to keep the longer startup times.
#3927 (comment)

@mutiev
Copy link

mutiev commented Aug 3, 2022

Have same problem with airflow initiating in docker.

airflow-init_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

docker restart solve problem

@Jafner
Copy link

Jafner commented Dec 29, 2022

This is the error I got when some of the containers in a compose stack were dependent on a network share that was not mounted.

My NAS was offline and when I tried to docker-compose up any stacks with NAS-dependent containers, they would time out.

Just had to bring my NAS back online.

@scippacercola
Copy link

In my case it was an anonymous volume to cause the problem:

volumes:
      - /usr/src/app

I had to bind the folder to a local directory.

d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
d3vilh added a commit to d3vilh/raspberry-gateway that referenced this issue Jul 29, 2023
@tan-yong-sheng
Copy link

Faced the same problem:

My github repo I wish to use Binder is https://github.com/tys203831/portfolio_analysis

Removing intermediate container 60b0dfe441eb
 ---> 390c5bbcebc0
{"aux": {"ID": "sha256:390c5bbcebc04842eef6c909e77c4bb16ad9965557724aedb38f32dbaecf3044"}}Successfully built 390c5bbcebc0
Successfully tagged gesiscss/binder-r2d-g5b5b759-tys203831-2dportfolio-5fanalysis-63d906:78e3d7b82dfe345bcb4c98620cbfbd88b53a2c00
Error during build: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.```

@Knight174
Copy link

+1, docker restart solve the problem

@ndeloof
Copy link
Contributor

ndeloof commented Aug 7, 2023

I'm closing this issue as obsolete: Docker Compose V1 has reached end-of-life and we are not accepting any more changes. Please try and reproduce your issue with Compose V2 and create a new issue or PR with the relevant Compose V2 information.

@patsevanton
Copy link

to summarize, how to fix this issue?

@glours
Copy link
Contributor

glours commented Nov 17, 2023

@patsevanton If you still have the issue with a recent version of Compose v2, open a new issue with a example to reproduce it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests