"docker pull" fails after 1.13.0-rc3-beta32 upgrade with "server misbehaving"

[jamshid]

Expected behavior

docker pull should work after upgrading to latest beta.

Actual behavior

$ docker pull openknowledge/openldap
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.65.1:53: server misbehaving

Information

Docker for Mac: version: 1.13.0-rc3-beta32 (9cf974f9e)
OS X: version 10.11.6 (build: 15G1108)
logs: /tmp/9D052C1E-09A7-4126-BC50-6FC7D513ED7C/20161207-173830.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Steps to reproduce the behavior

1. upgrade a previously working docker for mac to latest beta
2. i do not have any VPN enabled, but i am on my work wifi (normally docker for mac works fine)
3. Yup, definitely something about being on work wifi. It worked when I switched to my personal hotspot wifi. When I switch back to work wifi the "docker pull" error returns.
4. If it helps, this is what's in my docker for mac's /etc/resolv.conf when on work wifi and docker pull fails:

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
/ # cat /etc/resolv.conf 
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
domain hq.EXAMPLE.com
search tx.EXAMPLE.com hq.EXAMPLE.com qa.EXAMPLE.com
nameserver 192.168.65.1
# /etc/resolv.conf.tail can replace this line

[djs55]

In the DNS logs inside the diagnostics I see some strange behaviour. You have 2 upstream DNS servers configured-- Docker for Mac sends queries to both servers and then waits for the first response.

For names which do exist such as registry-1.docker.io, the upstream server sends back a "Standard query response, Refused". Unfortunately this response comes back quickly and is first.

For names which don't exist such as registry-1.docker.io.searchdomain, the upstream server sends back a "Standard query response, No such name".

I suspect one of your upstream DNS servers is configured to disallow recursion. Could you perform some experiments for me? Could you run scutil --dns to find the IPs of your upstream servers, and then try querying them individually on the host with commands like

$ dig @8.8.8.8 www.google.com

; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37997
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.			IN	A

;; ANSWER SECTION:
www.google.com.		92	IN	A	216.58.208.132

;; Query time: 41 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Dec 20 15:43:29 2016
;; MSG SIZE  rcvd: 48

I suspect that we should probably modify the DNS forwarder in Docker for Mac to ignore these "Refused" responses, since they aren't very useful (and crowd out useful responses)

[jamshid]

Thanks, @djs55, I think you figured it out! One of my two VPN (work) DNS servers does not resolve www.google.com, and responds to dig with WARNING: recursion requested but not available .
Is that a bug/misconfiguration I can complain to IT department about, or is this a common setup?

$ scutil --dns 
...
resolver #1
  search domain[0] : tx.example.com
  search domain[1] : qa.example.com
  search domain[2] : hq.example.com
  search domain[3] : ad.example.com
  nameserver[0] : 172.30.0.202
  nameserver[1] : 172.30.0.42
  if_index : 14 (utun4)
  flags    : Scoped, Request A records
Reachable
...
$  dig @172.30.0.202 www.google.com   # works
...
;; ANSWER SECTION:
www.google.com.		212	IN	A	173.194.219.106
...

$ dig @172.30.0.42 www.google.com   # fails
; <<>> DiG 9.8.3-P1 <<>> @172.30.0.42 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 26896
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;www.google.com.			IN	A

;; Query time: 29 msec

[djs55]

@jamshid I suspect it's a bug / oversight in the configuration of the upstream server. If your primary DNS server were to fail then I suspect other software would also have problems with your secondary server. I think it's worth escalating, just in case they aren't aware of it.

Having said that, I'm hoping to work on DNS in Docker for Mac over the next few betas, and will try to make it cope more gracefully with this situation. Thanks again for the report!

[djs55]

The DNS resolver in the master branch should now be robust to malfunctioning upstream servers. The code should be released in beta 37, due in a couple of weeks.

[yallop]

@jamshid, could you please confirm that this problem is fixed for you in the latest beta (39)?

[samoht]

The fix is now released on both the beta and stable channels so I am closing the issue. Please re-open (or open a new one) with a new diagnostic ID you still have that problem.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked