DNS lookup does not work with Docker in Docker

[mingfang]

Steps using OSX Maverick, Vagrant
1-Install Docker Vagrant file

git clone https://github.com/dotcloud/docker.git
cd docker
vagrant up
vagrant ssh

2-Install dind

sudo -i
apt-get install -y git
git clone https://github.com/jpetazzo/dind.git

3-Build and Run Dind

docker build -t dind .
docker run -privileged -lxc-conf="lxc.aa_profile=unconfined" -t -i dind

4-At this point ping google.com does not work

ping google.com
ping: unknown host

But pinging google's ip works

ping 74.125.22.102
PING 74.125.22.102 (74.125.22.102): 48 data bytes
56 bytes from 74.125.22.102: icmp_seq=0 ttl=61 time=32.766 ms

5-My /etc/resolv.conf

cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3

6-My NAT rules

iptables -t nat -nxvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3 packets, 188 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 3 packets, 188 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 MASQUERADE  all  --  *      *       10.0.0.0/16         !10.0.0.0/16         

Chain DOCKER (2 references)
    pkts      bytes target     prot opt in     out     source               destination       

I've been trying for weeks to get Openstack + Docker to work.
With the recent Havana release I'm very close.
But this DNS problem is a showstopper.
Please help.

[jpetazzo]

Can you show the output of ip addr ls on the VM? And in the d-in-d container?
Can you try to resolve using Google's public DNS resolvers? e.g. host www.google.com 8.8.8.8 ?
If it works, you could start the d-in-d container with -dns 8.8.8.8.
Alternatively, I hope that I'll be able to understand what's happening from the output of ip addr :-)

[mingfang]

From VM

ip addr ls

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:88:0c:a6 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fe88:ca6/64 scope link
valid_lft forever preferred_lft forever
3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 46:85:5a:b4:93:2a brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
inet6 fe80::4485:5aff:feb4:932a/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether e2:7c:b6:29:a4:d9 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 scope global docker0
inet6 fe80::d4e0:9dff:fe50:d9a1/64 scope link
valid_lft forever preferred_lft forever
8: vethh1NQ9j: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master docker0 state UP qlen 1000
link/ether e2:7c:b6:29:a4:d9 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e07c:b6ff:fe29:a4d9/64 scope link
valid_lft forever preferred_lft forever

From dind

ip addr ls

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 5a:0f:7f:32:62:52 brd ff:ff:ff:ff:ff:ff
inet 10.0.42.1/16 scope global docker0
inet6 fe80::580f:7fff:fe32:6252/64 scope link
valid_lft forever preferred_lft forever
77: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 62:19:49:e7:b3:95 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.39/16 brd 172.17.255.255 scope global eth0
inet6 fe80::6019:49ff:fee7:b395/64 scope link
valid_lft forever preferred_lft forever

Using Google’s public DNS fixes dns lookup, but looks like my VM’s port forwarding doesn’t work.
I’m going to trying VM bridge network to see if it makes a difference.

On Oct 19, 2013, at 3:14 PM, Jérôme Petazzoni notifications@github.com wrote:

Can you show the output of ip addr ls on the VM? And in the d-in-d container?
Can you try to resolve using Google's public DNS resolvers? e.g. host www.google.com 8.8.8.8 ?
If it works, you could start the d-in-d container with -dns 8.8.8.8.
Alternatively, I hope that I'll be able to understand what's happening from the output of ip addr :-)

—
Reply to this email directly or view it on GitHub.

[jpetazzo]

OK, I see the problem: in the VM, you have lxcbr0 on 10.0.3.1/24, which overlaps with docker0 in DinD, which is 10.0.42.1/16.

Solution 1: tear down lxcbr0 in the VM (and make sure that there is not iptables rule doing fancy things with 10.0.3.0/24).

Solution 2: setup Docker (in DinD) to use a different address for docker0. (Shameless self-promotion: this blog post might help!)

[mingfang]

Jérôme

It works!
Thanks for your help.

Just one more question. I have this in /etc/network/interfaces
auto lo
iface lo inet loopback

auto docker0
iface docker0 inet static
address 10.1.1.1
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0

However, docker0 is not up when I start a Docker container. I have to manually do a ifup -a.
Is there a way to the interface file to be loaded when a container starts up?

Thanks
—ming

On Oct 23, 2013, at 12:38 PM, Jérôme Petazzoni notifications@github.com wrote:

OK, I see the problem: in the VM, you have lxcbr0 on 10.0.3.1/24, which overlaps with docker0 in DinD, which is 10.0.42.1/16.

Solution 1: tear down lxcbr0 in the VM (and make sure that there is not iptables rule doing fancy things with 10.0.3.0/24).

Solution 2: setup Docker (in DinD) to use a different address for docker0. (Shameless self-promotion: this blog post might help!)

—
Reply to this email directly or view it on GitHub.

[jpetazzo]

Ah, unfortunately, ifup -a is (more or less indirectly) triggered by upstart (or the plain old init, on Debian)...
The easiest solution might be to use ifup -a (or to configure the interface manually in a custom script).
I should update my blog post to explain that!