When resolving the Docker Host /etc/hosts is used only after the DNS server

[mtneug]

Description

I'm trying to use the docker:edge-dind image and link it to a docker:edge container, but all Docker commands time out and end in an error. The DOCKER_HOST is set properly to tcp://docker:2375 and I'm able to resolve docker to the dind container IP. Using the IP directly works fine. After some search I found docker/for-mac#1302, which describes the same error. Docker seems to first ask the DNS server and only than use the /etc/hosts file. Sadly, in my environment the DNS server is able to resolve docker (plus search domain). However, I would expect that the hosts file has precedence.

Steps to reproduce the issue:

1. docker run --privileged --name docker -d docker:edge-dind
2. docker run -it --link docker docker:edge sh
3. docker version

Describe the results you received:

Docker resolves the IP using a DNS server, which returns a wrong IP. The command outputs an error after a timeout.

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:15:15 2017
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

Describe the results you expected:

Docker uses the /etc/hosts file to resolve the domain name to get the correct IP.

Output of docker version:

Client:
 Version:      17.07.0-ce-rc2
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   36ce605
 Built:        Mon Aug  7 23:43:03 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.07.0-ce-rc2
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   36ce605
 Built:        Mon Aug  7 23:48:34 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 12
 Running: 2
 Paused: 0
 Stopped: 10
Images: 39
Server Version: 17.07.0-ce-rc2
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 192b3389rd5w5h7xofpludtr6
 Is Manager: true
 ClusterID: kj6e2ug0n2d0fww991efqi4k7
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 192.168.65.2
 Manager Addresses:
  192.168.65.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3addd840653146c90a254301d6c3a663c7fd6429
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.41-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 7
Total Memory: 13.68GiB
Name: moby
ID: IV5F:MZJI:EVWO:YUWY:G37K:C777:KW7N:L5MW:BUVT:HLHF:CUVL:NDB7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 51
 Goroutines: 166
 System Time: 2017-08-17T12:32:07.53965267Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

The command output above is using Docker for Mac, but I first saw that error in an Linux environment.

[cpuguy83]

I cannot reproduce any issue here.

When using the docker image, it sets DOCKER_HOST to tcp://docker:2375.
The resolver itself has nothing to do with docker. In this case there isn't even a local dns server in the container like there is when you use a user-defined network.

I also cannot reproduce this error. Are you sure the dind container is running?

[mtneug]

@cpuguy83 thank you for the quick response.

Sorry, the description was not clear enough. Let's start over from where both containers are started and linked. Within the container using docker:edge I can run the following:

/ # cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.3      docker c0433f2f2259
172.17.0.4      11d7fbfd4052


/ # ping -c 2 docker
PING docker (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: seq=0 ttl=64 time=0.133 ms
64 bytes from 172.17.0.3: seq=1 ttl=64 time=0.086 ms

--- docker ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.086/0.109/0.133 ms


/ # echo $DOCKER_HOST
tcp://docker:2375

So it seems everything is connected properly. However, when running docker version it cannot connect and times out. This is because of the following (IPs and domain names changed):

/ # apk add bind-tools --no-cache


/ # cat /etc/resolv.conf
# Generated by NetworkManager
search some.domain
nameserver 1.1.1.1


/ # dig -q docker +search +short
2.2.2.2

So in my environment docker + seach.domain can be resolved by the DNS server (this is missing in your case). Now Docker, instead of using the IP from /etc/hosts, is using the IP it gets from the DNS server (here 2.2.2.2). With strace this becomes evident (run container with --privileged):

/ # apk add strace --no-cache


/ # strace -e connect -f docker version
strace: Process 18 attached
strace: Process 19 attached
strace: Process 20 attached
strace: Process 21 attached
strace: Process 22 attached
[pid    17] connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16 <unfinished ...>
[pid    20] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16 <unfinished ...>
[pid    17] <... connect resumed> )     = 0
[pid    20] <... connect resumed> )     = 0
[pid    17] connect(4, {sa_family=AF_INET, sin_port=htons(2375), sin_addr=inet_addr("2.2.2.2")}, 16) = -1 EINPROGRESS (Operation in progress)

I would instead expect Docker to prefer the IP from /etc/hosts.

[cpuguy83]

Ok I found the issue.
It has to do with the way go's resolver works.
Note that if you do GODEBUG=netdns=cgo docker version, it changes the lookup ordering to use file lookup first.

Ultimately this can be configured in /etc/nsswitch.conf, which the go resolver will adhere to.

[cpuguy83]

And sorry, the actual problem is with Alpine + musl not with go.

[cpuguy83]

Again, mistaken, it's not musl, it's go doing this. This is a statically compiled docker build, and wouldn't be using musl's resolver here.

	// If /etc/nsswitch.conf doesn't exist or doesn't specify any
  	// sources for "hosts", assume Go's DNS will work fine.
  	if os.IsNotExist(nss.err) || (nss.err == nil && len(srcs) == 0) {
  		if c.goos == "solaris" {
  			// illumos defaults to "nis [NOTFOUND=return] files"
  			return fallbackOrder
  		}
  		if c.goos == "linux" {
  			// glibc says the default is "dns [!UNAVAIL=return] files"
  			// http://www.gnu.org/software/libc/manual/html_node/Notes-on-NSS-Configuration-File.html.
  			return hostLookupDNSFiles
  		}
  		return hostLookupFilesDNS
  	}

[mtneug]

I can confirm that after changing the resolver to cgo Docker did consult the hosts file first. Changing the /etc/nsswitch.conf file and using the Go resolver also worked. So, should we just close this issue as a configuration problem?

[cpuguy83]

Yeah, the behavior is strange but seems to be intentional.

Thanks!