socket.timeout: timed out on compose push

[FernandoMiguel]

Description of the issue

python trace weh pushing to registry

Context information (for bug reports)

we use docker-compose to push build images via jenkins CI to aws ecr

time docker-compose -f .jenkins/docker-compose.images-base.1.yml push

Output of docker-compose version

docker-compose version 1.24.1, build 4667896
docker-py version: 3.7.3
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.1b  26 Feb 2019```

**Output of `docker version`**

Client: Docker Engine - Community
Version: 19.03.1
API version: 1.40
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:21:22 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:19:53 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683```

python version
Python 3.7.3

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

version: "3.7"

services:
  php70:
    image: ${AWS_ECR_ACCOUNT:-REDACTED}.dkr.ecr.eu-west-1.amazonaws.com/php:php70${COMMIT_ID}
    build:
      context: ../infrastructure-as-code/Docker/php
      dockerfile: Dockerfile-7.0

Steps to reproduce the issue

1. time docker-compose -f .jenkins/docker-compose.images-base.1.yml push

Observed result

python trace

Expected result

Pushing to ECR
Pushing php70 (REDACTED.dkr.ecr.eu-west-1.amazonaws.com/php70-6d7b27e)...
The push refers to repository [REDACTED.dkr.ecr.eu-west-1.amazonaws.com/php]
php70-6d7b27e: digest: sha256:0c15dbf77f8c98acfde6fad306908c8ce9a71be23bb060bb1b0a5738c4856298 size: 5123

Stacktrace / full error message

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 444, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.7/http/client.py", line 447, in read
    n = self.readinto(b)
  File "/usr/lib/python3.7/http/client.py", line 481, in readinto
    return self._readinto_chunked(b)
  File "/usr/lib/python3.7/http/client.py", line 576, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.7/http/client.py", line 544, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.7/http/client.py", line 504, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 71, in main
    command()
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 127, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 776, in push
    ignore_push_failures=options.get('--ignore-push-failures')
  File "/usr/local/lib/python3.7/dist-packages/compose/project.py", line 654, in push
    service.push(ignore_push_failures)
  File "/usr/local/lib/python3.7/dist-packages/compose/service.py", line 1235, in push
    stream_output(output, sys.stdout))
  File "/usr/local/lib/python3.7/dist-packages/compose/progress_stream.py", line 112, in get_digest_from_push
    for event in events:
  File "/usr/local/lib/python3.7/dist-packages/compose/progress_stream.py", line 25, in stream_output
    for event in utils.json_stream(output):
  File "/usr/local/lib/python3.7/dist-packages/compose/utils.py", line 61, in split_buffer
    for data in stream_as_text(stream):
  File "/usr/local/lib/python3.7/dist-packages/compose/utils.py", line 37, in stream_as_text
    for data in stream:
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 345, in _stream_helper
    data = reader.read(1)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 461, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 367, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

Additional information

OS version / distribution, docker-compose install method, etc.
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.04
Release: 19.04
Codename: disco

pip3 install
wheel
docker-compose
awscli

[IoanaStanescu]

This also happened to me quite a few times but always assumed it was related with the connection, the push is made from a VM running in US on GCE to a ECR in Europe, in general after a retry it worked. I suspect this issue might also be related to a timeout threshold in docker-compose or the docker engine.

[troytoman]

I am seeing a similar issue pushing from a EKS container to Docker Hub. Also seeing this in some system logs: "Not continuing with push after error: context canceled"

[nkazakov]

I had the same problem with docker-compose push
You can set environment variables with your values:

DOCKER_CLIENT_TIMEOUT: 180
COMPOSE_HTTP_TIMEOUT: 180

It helped me.

[brunosousadev]

@nkazakov Hello could you explain me better how to set the environment variables? my problem is with docker-compose up

[nkazakov]

@nkazakov Hello could you explain me better how to set the environment variables? my problem is with docker-compose up

@brunosousadev Hello. You should create file .env (or add lines to existed file) with this parameters:

DOCKER_CLIENT_TIMEOUT=180
COMPOSE_HTTP_TIMEOUT=180

Time in seconds.

Or you can set these parameters to global environment variables of your OS.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[jamiegau]

HOW I GOT AROUND THIS.

Ok, I had this issue on a Mac, current release, to a registry docker image running on a server at work.. Would keep getting stuck on a specific blob.
After restarting the docker on my Mac (As I was getting desperate), the problem went away. Go figure!!!!
I hope this this helps others..

[minhkhiemm]

If anyone has the same problem with the new update of MacOS Big Sur after update to a new release(cannot start docker or docker sad starting any time or run docker-compose up have the same issue mentioned above -> may be caused by disk corruption) you can try Clean/Purge data in docker desktop and then restart, it worked for me

[EmiyaYang]

If anyone has the same problem with the new update of MacOS Big Sur after update to a new release(cannot start docker or docker sad starting any time or run docker-compose up have the same issue mentioned above -> may be caused by disk corruption) you can try Clean/Purge data in docker desktop and then restart, it worked for me

docker system prune -a did the trick for me.