Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection

[uzokis]

Description

Getting a time-out when running docker login, docker run hello-world or docker pull <anything>
Issue is happening on Fedora 25 with docker 1.13.0

Steps to reproduce the issue:

1. Run docker run hello-world

Describe the results you received:

➜  ~ docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

Describe the results you expected:
I expect to be able to run the hello-world container, or pull any container by using the default commands.

Additional information you deem important (e.g. issue happens only occasionally):
The issue is happening always and i know docker was behaving correctly some while back.
I'm not sure if the issue started happening due to some package upgrade since i haven't been working with docker on a daily basis this last month.
I have tried it on 2 different networks, both wifi though.

Output of docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:58:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:58:06 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 16
 Running: 0
 Paused: 0
 Stopped: 16
Images: 136
Server Version: 1.13.0
Storage Driver: devicemapper
 Pool Name: data--vg-docker--thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 21.47 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 43.18 GB
 Data Space Total: 178.3 GB
 Data Space Available: 135.1 GB
 Metadata Space Used: 7.188 MB
 Metadata Space Total: 4.454 GB
 Metadata Space Available: 4.447 GB
 Thin Pool Minimum Free Space: 17.83 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.136 (2016-11-05)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.7-201.fc25.x86_64
Operating System: Fedora 25 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.58 GiB
Name: khaleesi
ID: CAFA:QAZR:G5EJ:QK6X:TJ5J:J5QQ:ZU24:DLPS:DVDL:6CSP:5N75:YUDD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
I've been trying for a few days to get it working again but basicly have read every thread google came up with, since some of them seem to be fixed by changing DNS settings, i've been toying around with anything related to that. (see below for more details)

I did notice that my ~/.docker directory is empty, i'm not sure if that is normal or i should expect additional configuration there?

I'm at least able to connect through curl:

➜  ~ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

I've changed my network settings to not use the company DNS but google DNS:
I changed my wifi connection at /etc/sysconfig/network-scripts/ifcfg-myconnection by setting these values:

PEERDNS=no
DNS1=8.8.8.8
IPV6_PEERDNS=no
DNS2=2001:4860:4860::8888
DNS3=2001:4860:4860::8844

Which results in:

➜  ~ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

I verified whether this works:

➜  ~ nslookup registry-1.docker.io 
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	registry-1.docker.io
Address: 52.72.139.208
Name:	registry-1.docker.io
Address: 54.236.131.143
Name:	registry-1.docker.io
Address: 52.201.90.157

I've explicitly added the 8.8.8.8 dns server in the systemd conf file:

➜  ~ cat /etc/systemd/system/docker.service.d/start.conf 
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/data--vg-docker--thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true --storage-opt=dm.basesize=20G --dns=8.8.8.8

I've also checked the resolv.conf file generated by systemd-resolved:

➜  ~ cat /var/run/systemd/resolve/resolv.conf 
# This file is managed by systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known DNS servers.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 8.8.8.8
nameserver 8.8.4.4

Lastly i added the -D flag when starting dockerd and executed docker run hello-world once, i added that log in attachment.
journalctl.txt

[uzokis]

Today there was an update in dnf which included docker-engine-1.13.1-1.fc25.x86_64
After installing the problem was resolved so closing this ticket...

[lvmodan]

Met the same issue. Is there a solution for this issue?

[TheSegfault]

Me too, I'm behind a corporate proxy on Windows 10, I configured docker proxy correctly btw but still timeout

[westlakem]

Same here

[ghost]

For windows 10, just mark DNS Server to fixed: 8.8.8.8

[umbruch]

all this did not help me. If you are in a company behinde a proxy use this guide, it did solve my problem
https://docs.docker.com/engine/admin/systemd/#manually-creating-the-systemd-unit-files

[videni]

@brunowerneck, fixed dns doesn't solve the problem , I have tried 2 window10 machines

[antonagestam]

Hitting the same issue.

Version 17.06.0-ce-mac18 (18433)
Channel: stable
d9b66511e0

$ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:31:53 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
$ docker pull nginx
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[alldev0825]

Please refer to this: https://docs.docker.com/engine/admin/systemd/#httphttps-proxy
I have resolved on Ubuntu 16.04 LTS

[jameskroy]

docker documentation talks about using systemctl not for as service
I had the issue solved it by adding the two lines in the /etc/default/docker and restarting the service.
export HTTP_PROXY=http://xxx.xxx.xxx.xxx:8080/
export HTTPS_PROXY=http://xxx.xxx.xxx.xxx:8080/

[nicholas-fwang]

@alldev0825 Thx your link!!

[PDRMAN]

@alldev0825 Yes, that link was very helpful. We were trying to build AWX (open source tower). It would hang trying to pull the postgress db images from the Docker public registry even though we set proxy in the shell and in the inventory filel. Only when we set it in the systmctl environment variables did it work!

[mreiche]

Please refer to this: https://docs.docker.com/engine/admin/systemd/#httphttps-proxy
I have resolved on Ubuntu 16.04 LTS

I already had proxy environment variables defined like
http_proxy, https_proxy, HTTP_PROXY and HTTPS_PROXY but docker did not accepted this, so I had to follow this explicit service configuration as well.