Pulling repositories fail with DNS resolve error

[dinamic]

Expected behavior

Pulling repositories should work.

Actual behavior

It does not work.

Information

Diagnostic ID: B56CB7AD-A17D-4781-80E8-0E526CDEE585
Docker for Mac: 1.12.0-a (Build 11213)
macOS: Version 10.11.6 (Build 15G31)
[OK] docker-cli
[OK] app
[OK] moby-syslog
[OK] disk
[OK] virtualization
[OK] system
[OK] menubar
[OK] osxfs
[OK] db
[OK] slirp
[OK] moby-console
[OK] logs
[OK] vmnetd
[OK] env
[OK] moby
[OK] driver.amd64-linux

Steps to reproduce

I have switched from docker beta 1.12.1 to 1.12.0 stable. I am not sure if this is relevant.

1. docker pull busybox:latest
2. you get an error like so:

latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/2b/2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749/data?Expires=1471560614&Signature=IlbbGvtqzRfwUP0ROQP714swP8XRhZry8RnjLqSHtdPXraCAH5OVpJum5xrtznyjnFducYQEMX0Qbuh-9ZUbNxITWcdtz36ymx7gixyPOg81hGjUCmBA0k49-2neydJ3Nw8lEpH4OSjgw0cOctVVfofc5oRxaHReODUtOqF6~X4_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 192.168.65.1:53: no such host

Opening the URL via cURL works just fine:

# curl -v "https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/2b/2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749/data?Expires=1471560614&Signature=IlbbGvtqzRfwUP0ROQP714swP8XRhZry8RnjLqSHtdPXraCAH5OVpJum5xrtznyjnFducYQEMX0Qbuh-9ZUbNxITWcdtz36ymx7gixyPOg81hGjUCmBA0k49-2neydJ3Nw8lEpH4OSjgw0cOctVVfofc5oRxaHReODUtOqF6~X4_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q"
*   Trying 54.239.168.68...
* Connected to dseasb33srnrn.cloudfront.net (54.239.168.68) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.cloudfront.net
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /registry-v2/docker/registry/v2/blobs/sha256/2b/2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749/data?Expires=1471560614&Signature=IlbbGvtqzRfwUP0ROQP714swP8XRhZry8RnjLqSHtdPXraCAH5OVpJum5xrtznyjnFducYQEMX0Qbuh-9ZUbNxITWcdtz36ymx7gixyPOg81hGjUCmBA0k49-2neydJ3Nw8lEpH4OSjgw0cOctVVfofc5oRxaHReODUtOqF6~X4_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q HTTP/1.1
> Host: dseasb33srnrn.cloudfront.net
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
< Content-Length: 1459
< Connection: keep-alive
< Date: Thu, 18 Aug 2016 22:29:22 GMT
< Last-Modified: Thu, 23 Jun 2016 23:24:56 GMT
< ETag: "02fdb6c0129a5c863fda5d2f8999feed"
< x-amz-version-id: snBnG5F_tdvYeVXX9fKE_xxeE2Uz6nn9
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 169
< X-Cache: Hit from cloudfront
< Via: 1.1 3f22ccb0670a6dfd48cff193a28e878c.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: rY1VSqxui1FcAmq3dVSAym8WuvHwx_-8xPeLYai59i1TbJerB0PM0w==
<
* Connection #0 to host dseasb33srnrn.cloudfront.net left intact
{"architecture":"amd64","config":{"Hostname":"55cd1f8f6e5b","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["sh"],"Image":"sha256:e732471cb81a564575aad46b9510161c5945deaf18e9be3db344333d72f0b4b2","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{}},"container":"764ef4448baa9a1ce19e4ae95f8cdd4eda7a1186c512773e56dc634dff208a59","container_config":{"Hostname":"55cd1f8f6e5b","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","#(nop) CMD [\"sh\"]"],"Image":"sha256:e732471cb81a564575aad46b9510161c5945deaf18e9be3db344333d72f0b4b2","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{}},"created":"2016-06-23T23:23:37.198943461Z","docker_version":"1.10.3","history":[{"created":"2016-06-23T23:23:36.73131105Z","created_by":"/bin/sh -c #(nop) ADD file:9ca60502d646bdd815bb51e612c458e2d447b597b95cf435f9673f0966d41c1a in /"},{"created":"2016-06-23T23:23:37.198943461Z","created_by":"/bin/sh -c #(nop) CMD [\"sh\"]","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:8ac8bfaff55af948c796026ee867448c5b5b5d9dd3549f4006d9759b25d4a893"]}}%

[dinamic]

The error message varies.

# docker run busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
docker: error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/2b/2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749/data?Expires=1471561466&Signature=M5NR0eqFvgSxtOzcUSK5cRllRmyDx0Ta3G1DdgnWdm3OaOij~wkSuxe4wuwZh5aOoz1VlCG4rntT4QTSlEFrpML1Wj5YI4vQ-AQJqVdSL2Nvg~um1vuNqN2rZNxudfagHi3RYq9wrEkCh-WRcj-2Rv5cEdbSdeDCO2-lE78PA1Q_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 192.168.65.3:53: cannot unmarshal DNS message.
See 'docker run --help'.

I have no clue what this address is 192.168.65.3 and I am positive it is not my DNS server.

My DNS is on 192.168.88.1:

# nslookup google.com
Server:     192.168.88.1
Address:    192.168.88.1#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.212.46

[dinamic]

Seems to be duplicate of moby/moby#24344. Probably something local got corrupted whenever I installed the docker beta rc and pulling out the stable hasn't fixed it.

[dinamic]

As a temporary workaround, I have set my DNS server to 8.8.8.8.

[kladkogex]

Google NS servers not work.

dig index.docker.io

; <<>> DiG 9.9.5-11ubuntu1-Ubuntu <<>> index.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51985
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 13, ADDITIONAL: 13

;; QUESTION SECTION:
;index.docker.io.		IN	A

;; ANSWER SECTION:
index.docker.io.	134	IN	CNAME	elb-io.us-east-1.aws.dckr.io.
elb-io.us-east-1.aws.dckr.io. 464 IN	CNAME	us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com.
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 14 IN A 52.87.47.61
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 14 IN A 52.200.132.201
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 14 IN A 34.200.194.233

;; AUTHORITY SECTION:
.			2832	IN	NS	b.root-servers.net.
.			2832	IN	NS	c.root-servers.net.
.			2832	IN	NS	d.root-servers.net.
.			2832	IN	NS	e.root-servers.net.
.			2832	IN	NS	f.root-servers.net.
.			2832	IN	NS	g.root-servers.net.
.			2832	IN	NS	h.root-servers.net.
.			2832	IN	NS	i.root-servers.net.
.			2832	IN	NS	j.root-servers.net.
.			2832	IN	NS	k.root-servers.net.
.			2832	IN	NS	l.root-servers.net.
.			2832	IN	NS	m.root-servers.net.
.			2832	IN	NS	a.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net.	83287	IN	A	192.228.79.201
c.root-servers.net.	83287	IN	A	192.33.4.12
d.root-servers.net.	26935	IN	A	199.7.91.13
e.root-servers.net.	26935	IN	A	192.203.230.10
f.root-servers.net.	26935	IN	A	192.5.5.241
g.root-servers.net.	83287	IN	A	192.112.36.4
h.root-servers.net.	83287	IN	A	198.97.190.53
i.root-servers.net.	26935	IN	A	192.36.148.17
j.root-servers.net.	50329	IN	A	192.58.128.30
k.root-servers.net.	83287	IN	A	193.0.14.129
l.root-servers.net.	83287	IN	A	199.7.83.42
m.root-servers.net.	11457	IN	A	202.12.27.33
a.root-servers.net.	62582	IN	A	198.41.0.4

;; Query time: 45 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Jun 06 10:47:51 EEST 2017
;; MSG SIZE  rcvd: 620

Message size = 620.

[drzraf]

Please reopen, THIS is the main issue, but unrelated to DNS.
The issue is bandwidth contention. When pulling (generally 3 in parallel), in many configuration (think low bandwidth, traffic shaping from the internet provider, ...), we can get timeout.
It usually manifest through a DNS error (when the next download can't resolve registry-1.docker.io anymore for example.

So I tried to hardcode registry-1.docker.io and auth.docker.io inside /etc/hosts to overcome this error:
dial tcp: lookup registry-1.docker.io on 8.8.4.4:53: read udp 192.168.1.35:54288->8.8.4.4:53: i/o timeout
I can go a bit further, but that's not enough. Since connectivity is bad, the next error is about contacting dseasb33srnrn.cloudfront.net (TCP/SSL):
net/http: TLS handshake timeout

Something MUST be done about image downloading, and the most obvious one is about:

• resume downloads where they stop (wget/curl can do that for decades)
  (not everyone live with San Francisco fiber)
• give an option to limit the number of parallel downloads
• give hints in the documentation about how to throttle downloads.

FYI, I first tried this:
trickle -vs -d 100 -u 20 docker pull ...
This does not work, because the daemon process is in charge of the download
Then running the daemon throttled:
trickle -vs -d 150 -u 40 /usr/bin/dockerd -D -p /var/run/docker.pid
which does not work either because of iptables/trickle
trickle -vs -d 150 -u 40 /usr/bin/dockerd -D --iptables=false -p /var/run/docker.pid
And this fail to actually throttle the bandwidth for an unknown reason (LD_PRELOAD hack over docker wouldn't work?)

Out of luck, I still can't download an image (I'm stuck around 250MB which I must always download again)!

We can recognize the importance of this issue to the absurdness of the situation it put the user into.

NB: it's not OS/X specific, and there are at least 10 duplicate/subsequent issues of this main issue.
moby/moby#4763

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked