Trouble with domain to container resolution on OS X #73

gijs · 2014-09-08T11:18:47Z

Hi,

I'm having trouble connecting to containers by hostname from the OS X side.

Is there anything I'm doing wrong here? Any help/ideas greatly appreciated!

Thx! Gijs

The following is what I know so far.

Running dig from the OS X terminal replies as such:

$ dig elasticsearch.dev.docker.dev

; <<>> DiG 9.8.3-P1 <<>> elasticsearch.dev.docker.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23145
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;elasticsearch.dev.docker.dev.  IN  A

;; ANSWER SECTION:
elasticsearch.dev.docker.dev. 18 IN A   172.17.0.13

;; Query time: 0 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Mon Sep  8 13:02:02 2014
;; MSG SIZE  rcvd: 90

And I can query the container using curl using the IP address:

$ curl 172.17.0.13:9200

which returns:

{
  "status" : 200,
  "name" : "Ogre",
  "version" : {
    "number" : "1.2.2",
    "build_hash" : "9902f08efc3ad14ce27882b991c4c56b920c9872",
    "build_timestamp" : "2014-07-09T12:02:32Z",
    "build_snapshot" : false,
    "lucene_version" : "4.8"
  },
  "tagline" : "You Know, for Search"
}

But using curl to query by domain...

$ curl http://elasticsearch.dev.docker.dev:9200

...gives:

curl: (6) Could not resolve host: elasticsearch.dev.docker.dev

Details:

CoreOS (alpha channel) on OS X Mavericks using Vagrant/Virtualbox (installed with Homebrew and Cask)
I've added 172.17.0.3 and 172.17.42.1 alongside the DHCP-provided IP's to the DNS Servers in OS X's System Preferences under Network > Advanced
The added DNS servers show up in /etc/resolv.conf:

#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain nens.local
nameserver 172.17.0.3
nameserver 172.17.42.1
nameserver 192.168.1.2
nameserver 192.168.1.4

Added a route to VirtualBox using $ sudo route -n add -net 172.17.0.0 10.2.0.10
Output from $ docker logs skydns:

[debug] 1410171742 skydock: received event (start) 467159a7239d7314a8fa675f3ec781f30a6d4c69590c22c43e66b20f9a4e16fd gijs/elasticsearch:1.0
[info] 1410171742 skydock: adding 467159a723 (elasticsearch) to skydns

Output from $ docker logs skydock:

2014/09/08 11:08:03 Received DNS Request for "elasticsearch.dev.docker.dev." from "10.2.0.1:61645"

Curl'ing the SkyDNS registry:

$ curl -XGET http://172.17.0.10:8080/skydns/services/

gives:

[
  {
    "UUID": "8e04b49ff8",
    "Name": "skydock",
    "Version": "skydock",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.11",
    "Port": 80,
    "TTL": 19,
    "Expires": "2014-09-08T10:37:55.252560655Z"
  },
  {
    "UUID": "3b24e79710",
    "Name": "skydns",
    "Version": "skydns",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.10",
    "Port": 53,
    "TTL": 19,
    "Expires": "2014-09-08T10:37:55.254102479Z"
  },
  {
    "UUID": "08707bf365",
    "Name": "pggis",
    "Version": "pggis",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.12",
    "Port": 49156,
    "TTL": 29,
    "Expires": "2014-09-08T10:38:05.14786093Z"
  },
  {
    "UUID": "467159a723",
    "Name": "elasticsearch",
    "Version": "elasticsearch",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.13",
    "Port": 49157,
    "TTL": 14,
    "Expires": "2014-09-08T10:37:49.989654756Z"
  }
]

Output from $ scutil --dns:

DNS configuration

resolver #1
  search domain[0] : docker.dev
  search domain[1] : dev
  search domain[2] : nens.local
  nameserver[0] : 172.17.0.3
  nameserver[1] : 172.17.42.1
  nameserver[2] : 192.168.1.2
  nameserver[3] : 192.168.1.4
  flags    : Request A records
  reach    : Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : docker.dev
  search domain[1] : dev
  search domain[2] : nens.local
  nameserver[0] : 172.17.0.3
  nameserver[1] : 172.17.42.1
  nameserver[2] : 192.168.1.2
  nameserver[3] : 192.168.1.4
  if_index : 4 (en0)
  flags    : Scoped, Request A records
  reach    : Reachable

Another thing I noticed is that whenever I open a website in a browser, something like this shows up in SkyDNS's logs:

2014/09/08 11:14:06 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:07 Received DNS Request for "en.wikipedia.org." from "10.2.0.1:50015"
2014/09/08 11:14:07 Updated Service TTL: 467159a723 30
2014/09/08 11:14:08 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:09 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:10 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:11 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"

...however, the website did continue to work (wikipedia in this case)

The text was updated successfully, but these errors were encountered:

asbjornenge · 2014-09-08T11:25:45Z

@gijs have a look at the OSX weirdness paragraphs here

Maybe the same?

gijs · 2014-09-08T11:34:05Z

@asbjornenge thx for pointing there. It's that article that got me started in the first place ;) Adding the DNS server to OS X Network Preferences didn't do the trick in my case though :(

May I ask what your Network Prefs settings look like? Mine looks like this at the moment:

thx

asbjornenge · 2014-09-08T11:39:39Z

@gijs yeah, looks correct to me... Did you try with only the skydns container ip? Don't know if I have any other tips I'm afraid. Everything looks correct... weird..

asbjornenge · 2014-09-08T11:42:00Z

@gijs oh, hang on... might it be that your vm can't talk to the network?

Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout

Can you dig @8.8.8.8 from the host? And also from within a container?

miekg · 2014-09-08T12:33:26Z

On 8 Sep 2014 12:42, "asbjornenge" notifications@github.com wrote:

@gijs oh, hang on... might it be that your vm can't talk to the network?

Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout

This is probably a red herring.

Can you dig @8.8.8.8 from the host? And also from within a container?

But this is still a good thing to check.

—
Reply to this email directly or view it on GitHub.

gijs · 2014-09-08T12:43:14Z

Hmmm something magical happened...
I was trying nslookup in OSX to see if that worked.
Then this showed:

$ nslookup elasticsearch.dev.docker.dev
Server:     172.17.42.1
Address:    172.17.42.1#53

Name:   elasticsearch.dev.docker.dev
Address: 172.17.0.13

And this showed in $ docker logs skydns:

2014/09/08 11:44:28 Received DNS Request for "elasticsearch.dev.docker.dev." from "10.2.0.1:60197"

And voila! It works in the browser in OS X... out of the blue...
(too early to be happy though, as I dont yet understand the details of this magic)

Now I'm curious if container-to-container dns is working. thx so far, I'll close this for now.

ps. I could ping and dig 8.8.8.8 both from within a container and from the CoreOS VM.. not sure about that error indeed.

gijs · 2014-09-08T12:48:49Z

I left it alone for an hour or so, and then the browser window that I opened didn't respond anymore.

Had to run $ nslookup elasticsearch.dev.docker.dev again, then the browser came back to life...

Something to do with timeouts/ttls? This is a bit fragile....

miekg · 2014-09-08T15:16:08Z

Strange....someone (maybe me or someone else needs to port skydns2 to
skydock.). I have got skydns(2) running for months on a public server with
no ill effect.
On 8 Sep 2014 13:48, "Gijs Nijholt" notifications@github.com wrote:

Hmmm... I left it alone for an hour or so, and then the browser window
that I opened didn't respond anymore.

Had to run $ nslookup elasticsearch.dev.docker.dev again, then the
browser came back to life...

Something to do with timeouts/ttls? This is a bit fragile....

—
Reply to this email directly or view it on GitHub
#73 (comment)
.

asbjornenge · 2014-10-13T12:33:13Z

A note on curl that just hit me today... curl by default uses (only?) ipv6. So I need to tell it to use ipv4 in order to get it working...

❯ curl elasticsearch.taghub.kiwi:9200   
curl: (6) Could not resolve host: elasticsearch.taghub.kiwi

❯ curl elasticsearch.taghub.kiwi:9200 -4
{
  "status" : 200,
  "name" : "Stegron"
  ...

asbjornenge · 2014-10-22T09:37:16Z

This whole issue seems to be related to how operating systems handle dns lookups. It seems they are always querying both A and AAAA records. How they handle the results can vary between OS and application. Especially when there are only A results and no AAAA results present some systems behave, in lack of a better word; surprisingly.

Now, if docker had full ipv6 support, we could populate those records and solve this problem. But for now we are in a pickle. Disabling ipv6 or prioritising ipv4 dns queries for all containers is a huge PITA.

I've been fiddling with my own dns server for discovery over here. I recently added a --ipv4-only flag that will return A records even for AAAA queries. This actually seems to solve the problem. I'm not a DNS expert, so I don't know if it's appropriate or not, but it does work! 🚀

@miekg is this totally bonkers? Or a valid workaround while waiting for ipv6...?

miekg · 2014-10-22T09:59:37Z

Pretty amazed that --ipv4-only works at all....

What could be an issue here is that skydns (I must double check this) returns an NXDOMAIN for AAAA, which means a A lookup is useless as the node does not exist. It should return NODATA so that a follow-up A lookup is done.

Is a possible to show 'dig A' and 'dig AAAA' responses here?

asbjornenge · 2014-10-22T10:07:14Z

@miekg AHA! That makes sense... I will try returning NODATA for the AAAA.

❯ dig postgresql.dance.kiwi A
// =>
;; QUESTION SECTION:
;postgresql.dance.kiwi.     IN  A

;; ANSWER SECTION:
postgresql.dance.kiwi.  5   IN  A   172.17.1.37


❯ dig "postgresql.dance.kiwi" AAAA
// =>
;; QUESTION SECTION:
;postgresql.dance.kiwi.     IN  AAAA

;; ANSWER SECTION:
postgresql.dance.kiwi.  5   IN  A   172.17.1.37

And none of the distributions I have tried are complaining anymore. Still, that NODATA is probably the correct fix. Thanks!

miekg · 2014-10-22T10:17:13Z

This being skydns(1) it should still do the right thing. I'm pretty sure skydns2 does so. Anyway I will work on getting skydock into skydns2.

asbjornenge · 2014-10-22T10:22:02Z

@miekg no sorry, this was rainbow-dns responses (with --ipv4-only). I don't have skydock set up anymore...

But a 👍 for gettings skydns2 into skydock

gijs closed this as completed Sep 8, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trouble with domain to container resolution on OS X #73

Trouble with domain to container resolution on OS X #73

gijs commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

gijs commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

miekg commented Sep 8, 2014

gijs commented Sep 8, 2014

gijs commented Sep 8, 2014

miekg commented Sep 8, 2014

asbjornenge commented Oct 13, 2014

asbjornenge commented Oct 22, 2014

miekg commented Oct 22, 2014

asbjornenge commented Oct 22, 2014

miekg commented Oct 22, 2014

asbjornenge commented Oct 22, 2014

Trouble with domain to container resolution on OS X #73

Trouble with domain to container resolution on OS X #73

Comments

gijs commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

gijs commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

asbjornenge commented Sep 8, 2014

miekg commented Sep 8, 2014

gijs commented Sep 8, 2014

gijs commented Sep 8, 2014

miekg commented Sep 8, 2014

asbjornenge commented Oct 13, 2014

asbjornenge commented Oct 22, 2014

miekg commented Oct 22, 2014

asbjornenge commented Oct 22, 2014

miekg commented Oct 22, 2014

asbjornenge commented Oct 22, 2014