New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dig dnsdocker.docker works, curl dnsdocker.docker doesn't #34

Closed
oliverwilkie opened this Issue Aug 4, 2015 · 20 comments

Comments

Projects
None yet
@oliverwilkie
Copy link

oliverwilkie commented Aug 4, 2015

Hi

From one container I am able to ping another container but any attempt to do a higher-level request such as curl fails citing that it 'could not resolve host'. Any theories/suggestions why? Both containers are based off Ubuntu 14.04

root@ac69fc055883:/app# dig dnsdock.docker

; <<>> DiG 9.9.5-3ubuntu0.4-Ubuntu <<>> dnsdock.docker
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35804
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;dnsdock.docker.            IN  A

;; ANSWER SECTION:
dnsdock.docker.     0   IN  A   172.17.3.48

;; Query time: 1 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Tue Aug 04 17:16:58 UTC 2015
;; MSG SIZE  rcvd: 62

But when I curl...

root@fab381760c0e:/app# strace -o /tmp/wtf -fF curl -v http://dnsdock.docker/services
* Hostname was NOT found in DNS cache
* Could not resolve host: dnsdock.docker
* Closing connection 0
curl: (6) Could not resolve host: dnsdock.docker

@oliverwilkie oliverwilkie reopened this Aug 4, 2015

@oliverwilkie

This comment has been minimized.

Copy link

oliverwilkie commented Aug 4, 2015

(accidentally posted wrong dig output, have corrected post)

@oliverwilkie

This comment has been minimized.

Copy link

oliverwilkie commented Aug 4, 2015

Note I can access http://dnsdock.docker/services from my host machine (Mac, Boot2Docker) having followed the README instructions of Mac OS X. I just can't get inter-container resolving to work.

@oliverwilkie

This comment has been minimized.

Copy link

oliverwilkie commented Aug 4, 2015

Update: curl -4 -v http://dnsdock.docker/services worked fine (IPv4)

@oliverwilkie

This comment has been minimized.

Copy link

oliverwilkie commented Aug 4, 2015

I think this is related to #4

I was using tonistiigi/dnsdock:latest

If I instead use the latest available tagged build tonistiigi/dnsdock:v1.10.0 it works as expected

@bersace

This comment has been minimized.

Copy link

bersace commented Aug 31, 2015

Hi, We hit the same issue with base debian wheezy image.

Actually, the problem is that glibc resolver queries AAAA and A records. dnsdock sends NXDOMAIN for AAAA. This lead to failure, even if A is present. This is the implementation of glibc. curl -4 ignores AAAA, even if glibc resolv queries them.

dnsdock image 0dae40984f82 works fine. But latest (at the time of this comment) fails. The only solution we found was to downgrade to 0dae40984f82.

@tonistiigi can you reproduce this bug with a debian container ? Maybe busybox uses another resolver (eglibc ?) that handle properly NXDOMAIN in AAAA while A is ok.

@bersace

This comment has been minimized.

Copy link

bersace commented Aug 31, 2015

You can view this error with strace and tcpdump -i docker0 port 53

@bersace

This comment has been minimized.

Copy link

bersace commented Aug 31, 2015

cc @toopy

@mpeters

This comment has been minimized.

Copy link

mpeters commented Sep 1, 2015

I'm also seeing this problem. It can easily be seen by using getent comparing the results of "hosts" vs "ahosts":

$ getent hosts test.docker

$ getent ahosts test.docker

"hosts" works, but "ahosts" doesn't. I can also confirm that dropping back to v1.10.0 from latest fixes the issue.

@diogogmt

This comment has been minimized.

Copy link

diogogmt commented Sep 4, 2015

I'm facing similar issues with the latest build.

In an ubuntu container:

It works with:

  • dig
  • ping

Doesn't work with:

  • curl
  • wget
  • nslookup

On my mac

It works with:

  • ping
  • dig
  • curl

Doesn't work with:

  • nslookup

If would help I could post some more detailed logs from the error scenarios.

@tonistiigi

This comment has been minimized.

Copy link
Collaborator

tonistiigi commented Sep 4, 2015

@kgutwin do you know whats wrong here? do we need to revert #32 ?

@bersace

This comment has been minimized.

Copy link

bersace commented Sep 4, 2015

Hmm, looks like we get false NXDOMAIN. Maybe just a fix of #32 to actually get NOERROR.

Note that we override service name with DNSDOCK_ALIAS. Maybe #32 doesn't handle this case ?

@kgutwin

This comment has been minimized.

Copy link
Collaborator

kgutwin commented Sep 8, 2015

First - those of you still seeing NXDOMAIN on AAAA queries, can you verify you've pulled the latest tonistiigi/dnsdock image? I don't get NXDOMAIN from the image 3b38edc92eb7.

root@244f9cf57b75:/# host -t AAAA dnsdock.docker
dnsdock.docker has no AAAA record

However, I was able to reproduce the reported curl failure on Ubuntu 14.04.3. From the tcpdump output:

16:59:11.997600 IP 172.18.0.3.50946 > 172.18.42.1.53: 16672+ A? dnsdock.docker. (32)
16:59:11.997894 IP 172.18.0.3.50946 > 172.18.42.1.53: 44373+ AAAA? dnsdock.docker. (32)
16:59:11.998397 IP 172.18.42.1.53 > 172.18.0.3.50946: 16672- 1/0/0 A 172.18.0.8 (62)
16:59:11.998617 IP 172.18.42.1.53 > 172.18.0.3.50946: 44373- 0/1/0 (110)

Everything seemed to be ok, except the minus sign next to the query ID number on the replies. I traced that down to the lack of the 'recursion available' bit set on the reply packets. You can see that also as the "recursion requested but not available" warning from dig.

I'm not sure exactly why, but Ubuntu's resolver seems to care if recursion is requested but not available. Adding that bit to all replies seems to resolve the problem - see the PR below.

@kaluzki

This comment has been minimized.

Copy link

kaluzki commented Sep 13, 2015

with both pull requests from https://github.com/ps2/dnsdock curl works again

@diogogmt

This comment has been minimized.

Copy link

diogogmt commented Sep 15, 2015

Tested and verified that with the latest build from @ps2 most tools are now able to resolve the names.
Works with:

  • ping
  • dig
  • curl
  • wget

Still nslookup can't resolve domains, it keeps coming back as a name collision error as 127.0.53.53

I enabled some logs on dnsdock.
Below is the trace when I issue a nslookup helionode.cloud query

HandleRequest - dns:
;; opcode: QUERY, status: NOERROR, id: 27895
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;helionode.cloud.   IN   A

---------
Setting reply
Cleaned query helionode.cloud
queryServices helionode.cloud
service &{helionode_test node-js-ubuntu 172.17.0.23 -1 [helionode.cloud]}
DNS Type 1
DNS TypeA, making service A - name helionode.cloud.
 service &{helionode_test node-js-ubuntu 172.17.0.23 -1 [helionode.cloud]}
makeServiceA - service &{helionode_test node-js-ubuntu 172.17.0.23 -1 [helionode.cloud]} - n helionode.cloud.
rr helionode.cloud. 0   IN  A   172.17.0.23
After making service A
Writting message ;; opcode: QUERY, status: NOERROR, id: 27895
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;helionode.cloud.   IN   A

;; ANSWER SECTION:
helionode.cloud.    0   IN  A   172.17.0.23
+



@diogogmt

This comment has been minimized.

Copy link

diogogmt commented Sep 25, 2015

@tonistiigi any updates on this issue?
looks like the pr from https://github.com/ps2/dnsdock solved the issue.

@diogogmt

This comment has been minimized.

Copy link

diogogmt commented Oct 4, 2015

Besides applying the patches from PRs #45 and #47 I also had to update the mikeg dns library to its latest version.
The forked version with the changes can be found here

Now I can resolve the container DNS names using all tools. I tested both in a mac an ubuntu environment.

@rijnhard

This comment has been minimized.

Copy link

rijnhard commented Dec 18, 2015

Its December already and published fixes for this haven't been merged into the repo? I don't mean to sound ungrateful, but if you aren't going to maintain this then at least let someone else, or say so.

I mean, I just wasted a day chasing what was wrong with this. Only to find this thread.

@nterray

This comment has been minimized.

Copy link

nterray commented Jan 28, 2016

I have the same issue. What is missing in order to merge PRs and fix it?

@eliasson86

This comment has been minimized.

Copy link

eliasson86 commented Jan 28, 2016

Me too would like to see a merge happen here.

@ps2

This comment has been minimized.

Copy link
Collaborator

ps2 commented Feb 13, 2016

Merged

@ps2 ps2 closed this Feb 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment