Docker for Mac 1.12.0 TCP DNS queries fail #24344

Closed
gmr opened this Issue Jul 5, 2016 · 31 comments

Comments

Projects
None yet
@gmr

gmr commented Jul 5, 2016

Since Docker for Mac 1.12.0, fragmented DNS queries cause DNS resolution to fail when trying to resolve via 192.168.65.1:53 because the resolver can not connect via TCP to said IP.

Output of docker version:

Client:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        Sat Jul  2 00:09:24 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   876f3a7
 Built:        Tue Jul  5 02:20:13 2016
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0-rc3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: 7NQP:NH7Q:G2HH:3TIS:3L6P:EPES:SS5I:YEIU:7XEB:ZU37:UX27:6RTI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 17
 Goroutines: 29
 System Time: 2016-07-05T20:39:53.984107317Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: gavinmroy
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
This only happens when the DNS result is truncated, as can happen with auth.docker.io.

Please see https://forums.docker.com/t/auth-docker-io-on-192-168-65-1-53-no-such-host/16801/4 - It's worth noting that there are multiple posts in the Docker for Mac forum about this issue.

I was hoping the forums were visible enough for reports there, but I'm guessing I should have gone here first.

Steps to reproduce the issue:

  1. docker pull alpine

Describe the results you received:
Docker fails to pull the image due to DNS failure:

$ docker pull alpine
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/library/alpine/manifests/latest1: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io: dial tcp: lookup auth.docker.io on 192.168.65.1:53: no such host

Describe the results you expected:
Docker pulls the image.

Additional information you deem important (e.g. issue happens only occasionally):

  • This has something to do with the EDNS record sizes returned for auth.docker.io and the new Alpine 3.4 based moby image. Previous versions of moby on my network do not have the issue
  • The error that I get is a failure to connect to the embedded DNS server in docker via TCP
    On my network, I am getting EDNS truncated responses back which should cause the DNS resolver to failover to TCP queries
  • Alpine 3.4 does not have a problem with truncated issues on its own, so I'm guessing that the main issue here is lack of full EDNS/TCP support in the embedded docker DNS server, which seems to cause an issue when issuing a docker login/pull or anything else asking to resolve auth.docker.io since the record seems to be longer than the docker embedded DNS server can handle.
  • When I am in the moby instance, I can not resolve auth.docker.io with the default settings:
moby:~# nslookup auth.docker.io
;; Truncated, retrying in TCP mode.
;; Connection to 192.168.65.1#53(192.168.65.1) for auth.docker.io failed: connection refused.

If I change the DNS server to 8.8.8.8, I do not have a problem:

moby:~# cat /etc/resolv.conf 
search local
nameserver 8.8.8.8
moby:~# nslookup auth.docker.io
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
auth.docker.io  canonical name = elb-registry.us-east-1.aws.dckr.io.
elb-registry.us-east-1.aws.dckr.io      canonical name = us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com.
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.73.165.108
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.203.219.86
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.71.80.248
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.172.251.194
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.71.245.229
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.164.225.120
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.72.94.105
Name:   us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.174.255.71

I'm not quite sure why the results are being truncated as the resolver for my laptop has a max-udp-size and edns-udp-size of 4096. What that is material to the cause of the truncated EDNS UDP packets, it is not as much of a problem as Docker's embedded DNS server running at 192.168.65.1 doesn't seem to be listening for DNS queries via TCP.

I would assume that if 192.168.65.1 was able to resolve DNS via TCP (which is the fallback behavior when a UDP DNS response returns a fragmented reply), it would be connectable on port 53:

moby:~# exit

Welcome to Moby alpha
Kernel 4.4.14-moby on an x86_64 (/dev/ttyS0)

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

moby login: root
Welcome to the Moby alpha, based on Alpine Linux.
moby:~# telnet 192.168.65.1 53
telnet: can't connect to remote host (192.168.65.1): Connection refused
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
Member

thaJeztah commented Jul 6, 2016

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Jul 6, 2016

Thanks for the very clear bug report!

I think this was broken in commit moby/vpnkit@a56356f

There's a potential fix in moby/vpnkit#72

djs55 commented Jul 6, 2016

Thanks for the very clear bug report!

I think this was broken in commit moby/vpnkit@a56356f

There's a potential fix in moby/vpnkit#72

@mavenugo

This comment has been minimized.

Show comment
Hide comment
@mavenugo

mavenugo Jul 6, 2016

Contributor

Thanks @djs55

Contributor

mavenugo commented Jul 6, 2016

Thanks @djs55

@chadlung

This comment has been minimized.

Show comment
Hide comment
@chadlung

chadlung Jul 6, 2016

FYI - Seeing this issue as well. Using version 1.12.0-rc3-beta18 (build: 9996).

chadlung commented Jul 6, 2016

FYI - Seeing this issue as well. Using version 1.12.0-rc3-beta18 (build: 9996).

@guilhermebr

This comment has been minimized.

Show comment
Hide comment
@guilhermebr

guilhermebr Jul 12, 2016

Same here.

I have 3 dns at resolv.conf, the third is 8.8.8.8, but only worked when I moved 8.8.8.8 to the first position.

$ docker pull postgres
Using default tag: latest
Pulling repository docker.io/library/postgres
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/postgres/images. You may want to check your internet connection or if you are behind a proxy.

Client:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 91e29e8
Built: Sat Jul 2 00:09:24 2016
OS/Arch: darwin/amd64
Experimental: true

Server:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 876f3a7
Built: Tue Jul 5 02:20:13 2016
OS/Arch: linux/amd64
Experimental: true

guilhermebr commented Jul 12, 2016

Same here.

I have 3 dns at resolv.conf, the third is 8.8.8.8, but only worked when I moved 8.8.8.8 to the first position.

$ docker pull postgres
Using default tag: latest
Pulling repository docker.io/library/postgres
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/postgres/images. You may want to check your internet connection or if you are behind a proxy.

Client:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 91e29e8
Built: Sat Jul 2 00:09:24 2016
OS/Arch: darwin/amd64
Experimental: true

Server:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 876f3a7
Built: Tue Jul 5 02:20:13 2016
OS/Arch: linux/amd64
Experimental: true

@dan-coates

This comment has been minimized.

Show comment
Hide comment
@dan-coates

dan-coates Jul 23, 2016

Is there any sense of when this fix might get released? I'm seeing this problem in rc4-beta20 today when trying to resolve the proxy host that will even allow me to hit docker.io.

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository docker.io/library/hello-world
docker: Error while pulling image: Get https://index.docker.io/v1/repositories/library/hello-world/images: http: error connecting to proxy [proxy-url]: dial tcp: lookup [proxy-host] on 192.168.65.1:53: no such host.

Is there any sense of when this fix might get released? I'm seeing this problem in rc4-beta20 today when trying to resolve the proxy host that will even allow me to hit docker.io.

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository docker.io/library/hello-world
docker: Error while pulling image: Get https://index.docker.io/v1/repositories/library/hello-world/images: http: error connecting to proxy [proxy-url]: dial tcp: lookup [proxy-host] on 192.168.65.1:53: no such host.
@mshahat

This comment has been minimized.

Show comment
Hide comment
@mshahat

mshahat Jul 23, 2016

Same here, I'm having the same problem today using rc4-beta20 on mac


➜ ~  mshahat docker login reldocker.tibco.com
Username: mshahat
Password: 
Error response from daemon: Get https://reldocker.tibco.com/v1/users/: dial tcp: lookup reldocker.tibco.com on 192.168.65.1:53: no such host

➜ ~  mshahat docker version
Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      linux/amd64
 Experimental: true

screen shot 2016-07-23 at 12 22 55

mshahat commented Jul 23, 2016

Same here, I'm having the same problem today using rc4-beta20 on mac


➜ ~  mshahat docker login reldocker.tibco.com
Username: mshahat
Password: 
Error response from daemon: Get https://reldocker.tibco.com/v1/users/: dial tcp: lookup reldocker.tibco.com on 192.168.65.1:53: no such host

➜ ~  mshahat docker version
Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      linux/amd64
 Experimental: true

screen shot 2016-07-23 at 12 22 55

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Jul 27, 2016

One scenario which we haven't completely fixed yet is when the first name server fails -- we will send UDP queries to the second server but not (yet) TCP queries. If bug happens again, could you check whether your name servers are currently accepting TCP queries? (As an example my local ISP only seems to support TCP on the first name server, and unfortunately this name server regularly goes down, causing transient glitches.)

On your Mac, for each name server listed in /etc/resolv.conf, try a command like this:

dig +tcp @<nameserver IP> <interesting name>

If possible try looking up the same name from inside a container, e.g.

docker run -it alpine sh
apk update
apk add drill
drill -t <interesting name> @192.168.65.1

(connections to 192.168.65.1 should be forwarded to your first name server. In future connections to 192.168.65.3 will be forwarded to your second name server)

djs55 commented Jul 27, 2016

One scenario which we haven't completely fixed yet is when the first name server fails -- we will send UDP queries to the second server but not (yet) TCP queries. If bug happens again, could you check whether your name servers are currently accepting TCP queries? (As an example my local ISP only seems to support TCP on the first name server, and unfortunately this name server regularly goes down, causing transient glitches.)

On your Mac, for each name server listed in /etc/resolv.conf, try a command like this:

dig +tcp @<nameserver IP> <interesting name>

If possible try looking up the same name from inside a container, e.g.

docker run -it alpine sh
apk update
apk add drill
drill -t <interesting name> @192.168.65.1

(connections to 192.168.65.1 should be forwarded to your first name server. In future connections to 192.168.65.3 will be forwarded to your second name server)

@ozbillwang

This comment has been minimized.

Show comment
Hide comment
@ozbillwang

ozbillwang Aug 15, 2016

Contributor

fixed 👍

$ docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
$ cat /etc/resolv.conf
search local
nameserver 192.168.65.1
nameserver 192.168.65.3

# change to 8.8.8.8
$ cat /etc/resolv.conf
search local
nameserver 8.8.8.8

No need restart docker engine, works fine now.

Contributor

ozbillwang commented Aug 15, 2016

fixed 👍

$ docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
$ cat /etc/resolv.conf
search local
nameserver 192.168.65.1
nameserver 192.168.65.3

# change to 8.8.8.8
$ cat /etc/resolv.conf
search local
nameserver 8.8.8.8

No need restart docker engine, works fine now.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 18, 2016

As long as you have a local image (debian for example) I think you can connect to moby with the following command.

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

ghost commented Aug 18, 2016

As long as you have a local image (debian for example) I think you can connect to moby with the following command.

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
@dinamic

This comment has been minimized.

Show comment
Hide comment
@dinamic

dinamic Aug 19, 2016

@SydOps, as a temporary workaround you can set your OS X to use Google's public DNS 8.8.8.8. Works for me. That's how I was able to get the alpine container. :)

@djs55, here's the info you requested:

docker version
Client:
 Version:      1.12.1-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7889dc7
 Built:        Fri Aug 12 18:35:53 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.1-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7889dc7
 Built:        Fri Aug 12 18:35:53 2016
 OS/Arch:      linux/amd64
 Experimental: true

Verifying the bug happens:

docker run -it alpine:3.2 sh
Unable to find image 'alpine:3.2' locally
3.2: Pulling from library/alpine
bfc185be0245: Pulling fs layer
docker: error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/49/4933271a21f1a3eb183cae296ce2f405c8e0852fb4c90eae577b430393d7ef36/data?Expires=1471566573&Signature=G6TdXQg1c-oPj0KuyrDtirqgM~clW-ohpzLOEBNij-Q8EvNtheaaxJWb4jZi~4fn07iWu1yddXwCMPnh23lRy6coXRpiyoEYT2AK6O7j57ewG~5QrK5G61TNSvmc-CzvXAIUwFpDw81WQSAzSoLt~VlwDc4HSxnxRY6fiKatZ2k_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 192.168.65.3:53: cannot unmarshal DNS message.
See 'docker run --help'.

Querying from OS X works:

# dig +tcp @192.168.88.1 google.com

; <<>> DiG 9.8.3-P1 <<>> +tcp @192.168.88.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52163
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;google.com.            IN  A

;; ANSWER SECTION:
google.com.     299 IN  A   216.58.212.14

;; AUTHORITY SECTION:
google.com.     27901   IN  NS  ns3.google.com.
google.com.     27901   IN  NS  ns2.google.com.
google.com.     27901   IN  NS  ns1.google.com.
google.com.     27901   IN  NS  ns4.google.com.

;; ADDITIONAL SECTION:
ns3.google.com.     45642   IN  A   216.239.36.10
ns2.google.com.     27901   IN  A   216.239.34.10
ns1.google.com.     27901   IN  A   216.239.32.10
ns4.google.com.     27901   IN  A   216.239.38.10

;; Query time: 670 msec
;; SERVER: 192.168.88.1#53(192.168.88.1)
;; WHEN: Fri Aug 19 03:10:20 2016
;; MSG SIZE  rcvd: 180

Going into the container:

# docker run -it alpine sh
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz

v3.4.3-3-g543f7af [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.2-11-g9b41a63 [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5968 distinct packages available
/ # apk add drill
(1/2) Installing ldns (1.6.17-r3)
(2/2) Installing drill (1.6.17-r3)
Executing busybox-1.24.2-r9.trigger
OK: 5 MiB in 13 packages
/ # drill -t google.com @192.168.65.1
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 24221
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4
;; QUESTION SECTION:
;; google.com.  IN  A

;; ANSWER SECTION:
google.com. 238 IN  A   216.58.212.14

;; AUTHORITY SECTION:
google.com. 27840   IN  NS  ns1.google.com.
google.com. 27840   IN  NS  ns4.google.com.
google.com. 27840   IN  NS  ns3.google.com.
google.com. 27840   IN  NS  ns2.google.com.

;; ADDITIONAL SECTION:
ns1.google.com. 27840   IN  A   216.239.32.10
ns4.google.com. 27840   IN  A   216.239.38.10
ns3.google.com. 45581   IN  A   216.239.36.10
ns2.google.com. 27840   IN  A   216.239.34.10

;; Query time: 5 msec
;; SERVER: 192.168.65.1
;; WHEN: Fri Aug 19 00:11:20 2016
;; MSG SIZE  rcvd: 180
/ #

dinamic commented Aug 19, 2016

@SydOps, as a temporary workaround you can set your OS X to use Google's public DNS 8.8.8.8. Works for me. That's how I was able to get the alpine container. :)

@djs55, here's the info you requested:

docker version
Client:
 Version:      1.12.1-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7889dc7
 Built:        Fri Aug 12 18:35:53 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.1-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7889dc7
 Built:        Fri Aug 12 18:35:53 2016
 OS/Arch:      linux/amd64
 Experimental: true

Verifying the bug happens:

docker run -it alpine:3.2 sh
Unable to find image 'alpine:3.2' locally
3.2: Pulling from library/alpine
bfc185be0245: Pulling fs layer
docker: error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/49/4933271a21f1a3eb183cae296ce2f405c8e0852fb4c90eae577b430393d7ef36/data?Expires=1471566573&Signature=G6TdXQg1c-oPj0KuyrDtirqgM~clW-ohpzLOEBNij-Q8EvNtheaaxJWb4jZi~4fn07iWu1yddXwCMPnh23lRy6coXRpiyoEYT2AK6O7j57ewG~5QrK5G61TNSvmc-CzvXAIUwFpDw81WQSAzSoLt~VlwDc4HSxnxRY6fiKatZ2k_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 192.168.65.3:53: cannot unmarshal DNS message.
See 'docker run --help'.

Querying from OS X works:

# dig +tcp @192.168.88.1 google.com

; <<>> DiG 9.8.3-P1 <<>> +tcp @192.168.88.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52163
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;google.com.            IN  A

;; ANSWER SECTION:
google.com.     299 IN  A   216.58.212.14

;; AUTHORITY SECTION:
google.com.     27901   IN  NS  ns3.google.com.
google.com.     27901   IN  NS  ns2.google.com.
google.com.     27901   IN  NS  ns1.google.com.
google.com.     27901   IN  NS  ns4.google.com.

;; ADDITIONAL SECTION:
ns3.google.com.     45642   IN  A   216.239.36.10
ns2.google.com.     27901   IN  A   216.239.34.10
ns1.google.com.     27901   IN  A   216.239.32.10
ns4.google.com.     27901   IN  A   216.239.38.10

;; Query time: 670 msec
;; SERVER: 192.168.88.1#53(192.168.88.1)
;; WHEN: Fri Aug 19 03:10:20 2016
;; MSG SIZE  rcvd: 180

Going into the container:

# docker run -it alpine sh
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz

v3.4.3-3-g543f7af [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.2-11-g9b41a63 [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5968 distinct packages available
/ # apk add drill
(1/2) Installing ldns (1.6.17-r3)
(2/2) Installing drill (1.6.17-r3)
Executing busybox-1.24.2-r9.trigger
OK: 5 MiB in 13 packages
/ # drill -t google.com @192.168.65.1
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 24221
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4
;; QUESTION SECTION:
;; google.com.  IN  A

;; ANSWER SECTION:
google.com. 238 IN  A   216.58.212.14

;; AUTHORITY SECTION:
google.com. 27840   IN  NS  ns1.google.com.
google.com. 27840   IN  NS  ns4.google.com.
google.com. 27840   IN  NS  ns3.google.com.
google.com. 27840   IN  NS  ns2.google.com.

;; ADDITIONAL SECTION:
ns1.google.com. 27840   IN  A   216.239.32.10
ns4.google.com. 27840   IN  A   216.239.38.10
ns3.google.com. 45581   IN  A   216.239.36.10
ns2.google.com. 27840   IN  A   216.239.34.10

;; Query time: 5 msec
;; SERVER: 192.168.65.1
;; WHEN: Fri Aug 19 00:11:20 2016
;; MSG SIZE  rcvd: 180
/ #
@tunix

This comment has been minimized.

Show comment
Hide comment
@tunix

tunix Aug 29, 2016

Hi,

We're having this issue while trying to login a private docker registry. Will this PR make it to 1.12.1? Will you please add roadmap label to this one? :)

tunix commented Aug 29, 2016

Hi,

We're having this issue while trying to login a private docker registry. Will this PR make it to 1.12.1? Will you please add roadmap label to this one? :)

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
Contributor

cpuguy83 commented Aug 29, 2016

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Aug 30, 2016

There have been some TCP DNS fixes in the beta channel -- could you give that a go and let me know if it helps? https://download.docker.com/mac/beta/Docker.dmg

Note that it's not always guaranteed to be able to switch between beta and stable channels because the container format could change on disk. To be safe I recommend backing up your ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 so you can restore it again afterwards.

djs55 commented Aug 30, 2016

There have been some TCP DNS fixes in the beta channel -- could you give that a go and let me know if it helps? https://download.docker.com/mac/beta/Docker.dmg

Note that it's not always guaranteed to be able to switch between beta and stable channels because the container format could change on disk. To be safe I recommend backing up your ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 so you can restore it again afterwards.

@ozbillwang

This comment has been minimized.

Show comment
Hide comment
@ozbillwang

ozbillwang Aug 30, 2016

Contributor

@djs55

Update docker to beta 24, Still the same error. Check /etc/resolve.conf, get longer list than beta 23.

$ docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
/ # cat /etc/resolv.conf
search local
nameserver 192.168.65.1
nameserver 192.168.65.10
nameserver 192.168.65.9
nameserver 192.168.65.8
nameserver 192.168.65.7
nameserver 192.168.65.6
nameserver 192.168.65.5
nameserver 192.168.65.4
nameserver 192.168.65.3
/ #
Contributor

ozbillwang commented Aug 30, 2016

@djs55

Update docker to beta 24, Still the same error. Check /etc/resolve.conf, get longer list than beta 23.

$ docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
/ # cat /etc/resolv.conf
search local
nameserver 192.168.65.1
nameserver 192.168.65.10
nameserver 192.168.65.9
nameserver 192.168.65.8
nameserver 192.168.65.7
nameserver 192.168.65.6
nameserver 192.168.65.5
nameserver 192.168.65.4
nameserver 192.168.65.3
/ #
@kog

This comment has been minimized.

Show comment
Hide comment
@kog

kog Sep 8, 2016

@djs55 - I can also reproduce this on what the about pane tells me is Version 1.12.1-beta25 (build: 11807) 158f2eefd23de930e87abae9c30ab3612b1e520a, freshly downloaded for testing.

[kog@taters ~]$ docker --version
Docker version 1.12.1, build 23cf638, experimental

[kog@taters ~]$ docker pull alpine
Using default tag: latest
Pulling repository docker.io/library/alpine
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/alpine/images. You may want to check your internet connection or if you are behind a proxy.

For completeness I did also verify that adding the Google Public DNS at 8.8.8.8 works IFF added as the first item in /etc/resolv.conf

kog commented Sep 8, 2016

@djs55 - I can also reproduce this on what the about pane tells me is Version 1.12.1-beta25 (build: 11807) 158f2eefd23de930e87abae9c30ab3612b1e520a, freshly downloaded for testing.

[kog@taters ~]$ docker --version
Docker version 1.12.1, build 23cf638, experimental

[kog@taters ~]$ docker pull alpine
Using default tag: latest
Pulling repository docker.io/library/alpine
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/alpine/images. You may want to check your internet connection or if you are behind a proxy.

For completeness I did also verify that adding the Google Public DNS at 8.8.8.8 works IFF added as the first item in /etc/resolv.conf

@shankie-san

This comment has been minimized.

Show comment
Hide comment
@shankie-san

shankie-san Sep 9, 2016

FWIW, a couple of my colleagues are having this problem, and it seems to have arisen after connecting to a WiFi network that had some dodgy DNS settings.

I can resolve the issue by, in Moby, editing /etc/resolv.conf and adding Google's DNS servers at the top, but this (obviously?) only works until the Docker daemon is restarted.

For us, adding Google's DNS on the host doesn't seem to work – it doesn't seem to get propagated through to Moby.

Are the namservers like...

nameserver 192.168.65.1
nameserver 192.168.65.10
nameserver 192.168.65.9

... internal to Moby?

FWIW, a couple of my colleagues are having this problem, and it seems to have arisen after connecting to a WiFi network that had some dodgy DNS settings.

I can resolve the issue by, in Moby, editing /etc/resolv.conf and adding Google's DNS servers at the top, but this (obviously?) only works until the Docker daemon is restarted.

For us, adding Google's DNS on the host doesn't seem to work – it doesn't seem to get propagated through to Moby.

Are the namservers like...

nameserver 192.168.65.1
nameserver 192.168.65.10
nameserver 192.168.65.9

... internal to Moby?

@chpio

This comment has been minimized.

Show comment
Hide comment
@chpio

chpio Sep 15, 2016

getting the same error on Arch Linux.

# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7
 Git commit:   23cf638
 Built:        Fri Aug 19 02:03:02 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7
 Git commit:   23cf638
 Built:        Fri Aug 19 02:03:02 2016
 OS/Arch:      linux/amd64

chpio commented Sep 15, 2016

getting the same error on Arch Linux.

# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7
 Git commit:   23cf638
 Built:        Fri Aug 19 02:03:02 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7
 Git commit:   23cf638
 Built:        Fri Aug 19 02:03:02 2016
 OS/Arch:      linux/amd64
@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Sep 20, 2016

Contributor

@gmr (and others) is this resolved for you with the latest beta or stable releases?

Contributor

justincormack commented Sep 20, 2016

@gmr (and others) is this resolved for you with the latest beta or stable releases?

@favoretti

This comment has been minimized.

Show comment
Hide comment
@favoretti

favoretti Sep 21, 2016

Hey @justincormack nope, still happening here.
Basically for me the moby image doesn't get DNS servers that are assigned by my VPN client, and fails to resolve private registries. When I log in to moby and add the private DNS servers - everything works.

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7.1
 Git commit:   6f9534c
 Built:        Thu Sep 15 11:20:26 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 17:32:24 2016
 OS/Arch:      linux/amd64
 Experimental: true
# docker login -u vlazarenko -p doesnotmatter https://registry.ecg.so/v1/
Error response from daemon: Get https://registry.ecg.so/v1/users/: dial tcp: lookup registry.ecg.so on 192.168.65.1:53: no such host

favoretti commented Sep 21, 2016

Hey @justincormack nope, still happening here.
Basically for me the moby image doesn't get DNS servers that are assigned by my VPN client, and fails to resolve private registries. When I log in to moby and add the private DNS servers - everything works.

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.7.1
 Git commit:   6f9534c
 Built:        Thu Sep 15 11:20:26 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 17:32:24 2016
 OS/Arch:      linux/amd64
 Experimental: true
# docker login -u vlazarenko -p doesnotmatter https://registry.ecg.so/v1/
Error response from daemon: Get https://registry.ecg.so/v1/users/: dial tcp: lookup registry.ecg.so on 192.168.65.1:53: no such host
@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Sep 21, 2016

Contributor

@djs55 are there still known issues here?

Contributor

justincormack commented Sep 21, 2016

@djs55 are there still known issues here?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 2, 2016

I still have this issue on MacOS Sierra. I can sometimes get around the problem by just restarting docker, but it happens regularly.

$ docker version
Client:
 Version:      1.12.2-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   45bed2c
 Built:        Tue Sep 27 23:38:15 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.2-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   45bed2c
 Built:        Tue Sep 27 23:38:15 2016
 OS/Arch:      linux/amd64
 Experimental: true

ghost commented Oct 2, 2016

I still have this issue on MacOS Sierra. I can sometimes get around the problem by just restarting docker, but it happens regularly.

$ docker version
Client:
 Version:      1.12.2-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   45bed2c
 Built:        Tue Sep 27 23:38:15 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.2-rc1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   45bed2c
 Built:        Tue Sep 27 23:38:15 2016
 OS/Arch:      linux/amd64
 Experimental: true
@ahfeel

This comment has been minimized.

Show comment
Hide comment
@ahfeel

ahfeel Oct 11, 2016

I have the same issue as @SydOps and @shankiesan, having a /etc/resolv.conf populated of a dozen of internal IPs, and DNS resolution taking ages. Have to manually change the resolv.conf of my container to get it working... I'm on 1.12.2-rc1 on OSX

ahfeel commented Oct 11, 2016

I have the same issue as @SydOps and @shankiesan, having a /etc/resolv.conf populated of a dozen of internal IPs, and DNS resolution taking ages. Have to manually change the resolv.conf of my container to get it working... I'm on 1.12.2-rc1 on OSX

@shankie-san

This comment has been minimized.

Show comment
Hide comment
@shankie-san

shankie-san Oct 11, 2016

FWIW @ahfeel it's working for us in the latest stable version...

FWIW @ahfeel it's working for us in the latest stable version...

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Oct 11, 2016

Here's a quick update on where we are (as of beta 28, due to be released this week)

What should work:

  • fallback to TCP should work using the same server-selection logic as UDP
  • the Mac's "System Preferences -> Network -> Advanced -> DNS" server list should be used as upstream servers (with caveats, see below). This uses the system config database rather than the legacy /etc/resolv.conf file on the host.
  • search domain configuration is forwarded to the VM on start (which does also mean an application restart is needed to change it)

What we're still working on:

  • if there are multiple upstream DNS servers (especially if more than 3) then the wrong servers are sometimes queried and this can cause lookup failures.
  • configurations where queries for a particular domain (e.g. *.corp) are sent to specific servers

Note the /etc/resolv.conf in the file contains multiple virtual DNS IP addresses, but these are mapped onto the servers in "System Preferences -> Network -> Advanced -> DNS". If you change the system preferences settings then it should affect lookups in the VM, even though the /etc/resolv.conf file doesn't change.

I'm hoping to address some of the remaining issues over the next few beta cycles -- I'll keep you posted. Thanks for your reports and your patience!

djs55 commented Oct 11, 2016

Here's a quick update on where we are (as of beta 28, due to be released this week)

What should work:

  • fallback to TCP should work using the same server-selection logic as UDP
  • the Mac's "System Preferences -> Network -> Advanced -> DNS" server list should be used as upstream servers (with caveats, see below). This uses the system config database rather than the legacy /etc/resolv.conf file on the host.
  • search domain configuration is forwarded to the VM on start (which does also mean an application restart is needed to change it)

What we're still working on:

  • if there are multiple upstream DNS servers (especially if more than 3) then the wrong servers are sometimes queried and this can cause lookup failures.
  • configurations where queries for a particular domain (e.g. *.corp) are sent to specific servers

Note the /etc/resolv.conf in the file contains multiple virtual DNS IP addresses, but these are mapped onto the servers in "System Preferences -> Network -> Advanced -> DNS". If you change the system preferences settings then it should affect lookups in the VM, even though the /etc/resolv.conf file doesn't change.

I'm hoping to address some of the remaining issues over the next few beta cycles -- I'll keep you posted. Thanks for your reports and your patience!

@favoretti

This comment has been minimized.

Show comment
Hide comment
@favoretti

favoretti Oct 11, 2016

@djs55 There's another use-case to this (although I'm not sure how relevant it is to this particular issue). If something (say, VPN client) updates /etc/resolv.conf on host machine - docker is blissfully unaware of that change. Currently I'm "fixing" this issue by just logging into moby and forcing VPN DNS as first in the list, but would be awesome it it would poll for changes on a regular basis.

@djs55 There's another use-case to this (although I'm not sure how relevant it is to this particular issue). If something (say, VPN client) updates /etc/resolv.conf on host machine - docker is blissfully unaware of that change. Currently I'm "fixing" this issue by just logging into moby and forcing VPN DNS as first in the list, but would be awesome it it would poll for changes on a regular basis.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Oct 12, 2016

Contributor

@favoretti this should work, modulo the issues that @djs55 mentioned - the DNS servers on Moby are just proxies to the Mac, which will use the latest information. If you have a specific feature request/issue around that that is not working, I recommend opening an issue on https://github.com/docker/for-mac/issues with details of exactly how to replicate.

Contributor

justincormack commented Oct 12, 2016

@favoretti this should work, modulo the issues that @djs55 mentioned - the DNS servers on Moby are just proxies to the Mac, which will use the latest information. If you have a specific feature request/issue around that that is not working, I recommend opening an issue on https://github.com/docker/for-mac/issues with details of exactly how to replicate.

@mitsuhiko

This comment has been minimized.

Show comment
Hide comment
@mitsuhiko

mitsuhiko Nov 21, 2016

I think I am hitting the same thing here but I'm not entirely sure yet about the mechanics. I noticed that if I put 8.8.8.8 as the DNS server into my mac (docker for mac) the VM can resolve just fine. If I use my router's IP which runs a DNS server the VM fails to resolve.

I think I am hitting the same thing here but I'm not entirely sure yet about the mechanics. I noticed that if I put 8.8.8.8 as the DNS server into my mac (docker for mac) the VM can resolve just fine. If I use my router's IP which runs a DNS server the VM fails to resolve.

@mashaalmemon

This comment has been minimized.

Show comment
Hide comment
@mashaalmemon

mashaalmemon Dec 17, 2016

Hi There,

Are there any updates on this issue?

I am having a similar experience running Docker 1.12.3 for Mac. DNS resolution does not seem to be reliable. May work for some period of time but then becomes intermitent. Restart of docker seems to fix the issue.

For now, I have just changed DNS servers to Google's as @mitsuhiko suggested. Seems to work for the moment but not sure if this is because of the restart required to action his suggestion (which I would do in any case to get DNS working) or the different DNS servers.

Hi There,

Are there any updates on this issue?

I am having a similar experience running Docker 1.12.3 for Mac. DNS resolution does not seem to be reliable. May work for some period of time but then becomes intermitent. Restart of docker seems to fix the issue.

For now, I have just changed DNS servers to Google's as @mitsuhiko suggested. Seems to work for the moment but not sure if this is because of the restart required to action his suggestion (which I would do in any case to get DNS working) or the different DNS servers.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Dec 17, 2016

Contributor

@mashaalmemon there will not be any more updates here, please file an issue in https://github.com/docker/for-mac/issues where your issue can be investigated with more details. I am going to close this issue as this is not a docker engine issue but Docker for Mac specific.

Contributor

justincormack commented Dec 17, 2016

@mashaalmemon there will not be any more updates here, please file an issue in https://github.com/docker/for-mac/issues where your issue can be investigated with more details. I am going to close this issue as this is not a docker engine issue but Docker for Mac specific.

@favoretti

This comment has been minimized.

Show comment
Hide comment

@mashaalmemon I logged docker/for-mac#997 for the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment