Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

container dns name resolution does not work when connecting a running container to an overlay network #19376

Closed
sebi-hgdata opened this issue Jan 15, 2016 · 6 comments
Closed

container dns name resolution does not work when connecting a running container to an overlay network #19376

sebi-hgdata opened this issue Jan 15, 2016 · 6 comments
Assignees
@mavenugo
Labels
area/networking kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. priority/P0 Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed.
Milestone
1.10.0

Comments

@sebi-hgdata
Copy link

Seems like dns name resolution does not work for containers that were connected to an overlay network using network connect. See last docker exec ping commands... t1 ping to t2 does not work, t1 ping to t2's ip works, t3 ping to t4 works, and t4 to t2 works

$ sudo docker version
Client:
 Version:      1.10.0-dev
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   b63b43d
 Built:        Fri Jan 15 18:13:29 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0-dev
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   b63b43d
 Built:        Fri Jan 15 18:13:29 2016
 OS/Arch:      linux/amd64


$ sudo docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 1060
Server Version: 1.10.0-dev
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge overlay
Kernel Version: 4.3.0-040300-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.706 GiB
Name: seb
ID: B2M7:KYAT:FVQA:TA6R:FLAN:DUF5:224N:JFD3:E4UF:KL2Y:FHDM:FPCI
Username: sebihgdata
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Cluster store: consul://localhost:8500
Cluster advertise: 192.168.0.106:2375

Stept to reproduce:
  $ ps -aef|grep consul
seb      12849     1  0 00:06 ?        00:00:00 /bin/sh -e -c /home/seb/hgdata/deployments/consul/consul agent -ui-dir=/home/seb/hgdata/deployments/consul/web_ui -config-dir=/home/seb/hgdata/deployments/consul/config/server -data-dir=/home/seb/hgdata/deployments/consul/data -bootstrap-expect=1 -node=ldap -client 0.0.0.0 /bin/sh
seb      12851 12849  0 00:06 ?        00:00:12 /home/seb/hgdata/deployments/consul/consul agent -ui-dir=/home/seb/hgdata/deployments/consul/web_ui -config-dir=/home/seb/hgdata/deployments/consul/config/server -data-dir=/home/seb/hgdata/deployments/consul/data -bootstrap-expect=1 -node=ldap -client 0.0.0.0
root     12928     1  0 00:06 ?        00:00:10 /usr/bin/docker daemon --userland-proxy=false --cluster-advertise=wlan0:2375 --cluster-store consul://localhost:8500 --tlsverify --tlscacert=/etc/ssl/docker/ca.cert --tlscert=/etc/ssl/docker/swarm.cert --tlskey=/etc/ssl/docker/swarm.key -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --bip=172.17.42.1/16 -s overlay -g /home/seb/hgdata/deployments/docker

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS         PORTS               NAMES
$ sudo docker network ls
NETWORK ID          NAME                DRIVER
f3f234e64483        none                null                
54d0981a7276        host                host                
2d314823bc90        bridge              bridge 

$ sudo docker network create -d overlay test
a733d9ca6b82ca322873c4437e207c62562d6b0a0bdd3d67225b7c8b8792b467

$ sudo docker run -td --name t1  ubuntu:14.04.2
5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27
seb@seb:~/hgdata/sfdc-ops/scripts$ sudo docker run -td --name t2  ubuntu:14.04.2
916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404
seb@seb:~/hgdata/sfdc-ops/scripts$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
916463ea787d        ubuntu:14.04.2      "/bin/bash"         4 seconds ago       Up 3 seconds                            t2
5cd211d06322        ubuntu:14.04.2      "/bin/bash"         11 seconds ago      Up 9 seconds                            t1

s$ sudo docker network connect test t1
s$ sudo docker network connect test t2
s$ sudo docker inspect t1
[
    {
        "Id": "5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27",
        "Created": "2016-01-15T23:04:45.962188979Z",
        "Path": "/bin/bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 13881,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-01-15T23:04:46.318715679Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:44ae5d2a191e49f4a4de97a8b3e50049333d64e283bb2aa397815bbe01398a24",
        "ResolvConfPath": "/home/seb/hgdata/deployments/docker/containers/5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27/resolv.conf",
        "HostnamePath": "/home/seb/hgdata/deployments/docker/containers/5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27/hostname",
        "HostsPath": "/home/seb/hgdata/deployments/docker/containers/5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27/hosts",
        "LogPath": "/home/seb/hgdata/deployments/docker/containers/5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27/5cd211d0632235593bc46b23239edb2706774a9071d91d36bcae0652910cdd27-json.log",
        "Name": "/t1",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "ShmSize": 67108864,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "KernelMemory": 0,
            "Memory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/seb/hgdata/deployments/docker/overlay/f65d74052b899b5fb67ecac8d7d77afdd9f08e15e02df9a3056b0b5a223f4e60/root",
                "MergedDir": "/home/seb/hgdata/deployments/docker/overlay/b42e65b0e6984c6a294ee138ee56a6b1cd171a36f781e08d07969021debd09e9/merged",
                "UpperDir": "/home/seb/hgdata/deployments/docker/overlay/b42e65b0e6984c6a294ee138ee56a6b1cd171a36f781e08d07969021debd09e9/upper",
                "WorkDir": "/home/seb/hgdata/deployments/docker/overlay/b42e65b0e6984c6a294ee138ee56a6b1cd171a36f781e08d07969021debd09e9/work"
            }
        },
        "Mounts": [],
        "Config": {
            "Hostname": "5cd211d06322",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "ubuntu:14.04.2",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a067542b620dd620c2acce837292360ee608bbd7e063a0ae9cbc166857fdae06",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/a067542b620d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "8341bc410674479091acfb85788e97761746ee2d521e301754361dc4a8a749b2",
            "Gateway": "172.17.42.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.1",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:01",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2d314823bc90a2c2f264cc9fe03bdcd5173887e82ca38de7d98bfc410167ec68",
                    "EndpointID": "8341bc410674479091acfb85788e97761746ee2d521e301754361dc4a8a749b2",
                    "Gateway": "172.17.42.1",
                    "IPAddress": "172.17.0.1",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:01"
                },
                "test": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "a733d9ca6b82ca322873c4437e207c62562d6b0a0bdd3d67225b7c8b8792b467",
                    "EndpointID": "f671305850fe6469c584f35267635e19517ed5f940aa2289e1c8b2edaecc6b8a",
                    "Gateway": "",
                    "IPAddress": "10.0.0.2",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:02"
                }
            }
        }
    }
]

$ sudo docker inspect t2
[
    {
        "Id": "916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404",
        "Created": "2016-01-15T23:04:52.130929748Z",
        "Path": "/bin/bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 13949,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-01-15T23:04:52.655899654Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:44ae5d2a191e49f4a4de97a8b3e50049333d64e283bb2aa397815bbe01398a24",
        "ResolvConfPath": "/home/seb/hgdata/deployments/docker/containers/916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404/resolv.conf",
        "HostnamePath": "/home/seb/hgdata/deployments/docker/containers/916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404/hostname",
        "HostsPath": "/home/seb/hgdata/deployments/docker/containers/916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404/hosts",
        "LogPath": "/home/seb/hgdata/deployments/docker/containers/916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404/916463ea787d3c1b5632b866f9e5bb2701ac8eb3d24726795f3f77bfc456f404-json.log",
        "Name": "/t2",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "ShmSize": 67108864,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "KernelMemory": 0,
            "Memory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/seb/hgdata/deployments/docker/overlay/f65d74052b899b5fb67ecac8d7d77afdd9f08e15e02df9a3056b0b5a223f4e60/root",
                "MergedDir": "/home/seb/hgdata/deployments/docker/overlay/5f442255e2296e5043cc9093076b98aa41af83d7c26eb720a3cc66dbc50c2da3/merged",
                "UpperDir": "/home/seb/hgdata/deployments/docker/overlay/5f442255e2296e5043cc9093076b98aa41af83d7c26eb720a3cc66dbc50c2da3/upper",
                "WorkDir": "/home/seb/hgdata/deployments/docker/overlay/5f442255e2296e5043cc9093076b98aa41af83d7c26eb720a3cc66dbc50c2da3/work"
            }
        },
        "Mounts": [],
        "Config": {
            "Hostname": "916463ea787d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "ubuntu:14.04.2",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "4627eb48ed967b578761b086e13946aaf657cc0f012719debcd5c8199acf9da4",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/4627eb48ed96",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "78d2fd173d473e563ac5ee81f5ed90b8c0b8c3525658fe5454ff595eb52496d6",
            "Gateway": "172.17.42.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2d314823bc90a2c2f264cc9fe03bdcd5173887e82ca38de7d98bfc410167ec68",
                    "EndpointID": "78d2fd173d473e563ac5ee81f5ed90b8c0b8c3525658fe5454ff595eb52496d6",
                    "Gateway": "172.17.42.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                },
                "test": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "a733d9ca6b82ca322873c4437e207c62562d6b0a0bdd3d67225b7c8b8792b467",
                    "EndpointID": "e4538ae95ce05de37c284012ba66383552cb8705ed47ef2802ad00220e291608",
                    "Gateway": "",
                    "IPAddress": "10.0.0.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:03"
                }
            }
        }
    }
]

$ sudo docker exec t1 ping -c 1 t2
ping: unknown host t2
$ sudo docker exec t2 ping -c 1 t1
ping: unknown host t1

$ sudo docker exec t1 ping -c 1 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.101 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.101/0.101/0.101/0.000 ms

s$ sudo docker run -td --name t3 --net=test ubuntu:14.04.2
d5642e64a0e8cb3b5c272849167032287518259b5b17b9538f35f167284abe08
seb@seb:~/hgdata/sfdc-ops/scripts$ 
seb@seb:~/hgdata/sfdc-ops/scripts$ sudo docker run -td --name t4 --net=test ubuntu:14.04.2
f6513bcc30588a5e184b585c6b079f1b3c44948edde2286c11a36f0fbc5d7e4e

sudo docker exec t3 ping -c 1 t4
PING t4 (10.0.0.5) 56(84) bytes of data.
64 bytes from t4.test (10.0.0.5): icmp_seq=1 ttl=64 time=0.085 ms

--- t4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.085/0.085/0.085/0.000 ms

$ sudo docker exec t4 ping -c 1 t2
PING t2 (10.0.0.3) 56(84) bytes of data.
64 bytes from t2.test (10.0.0.3): icmp_seq=1 ttl=64 time=0.106 ms

--- t2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.106/0.106/0.000 ms
@thaJeztah
Copy link
Member

Thanks for reporting!

ping @mavenugo @sanimej could you have a look?

@thaJeztah thaJeztah added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. area/networking labels Jan 16, 2016
@thaJeztah thaJeztah added this to the 1.10.0 milestone Jan 16, 2016
@thaJeztah
Copy link
Member

Haven't tried to reproduce yet, but adding this to the 1.10 milestone so we don't loose track

@mavenugo
Copy link
Contributor

thanks @sebi-hgdata. I could reproduce the issue & I think we know the root-cause as well.
We will get this resolved for the next RC.

@sanimej
Copy link

sanimej commented Jan 16, 2016

@thaJeztah @sebi-hgdata Yes, we noticed this issue. Will fix it in the next RC.

@thaJeztah thaJeztah added the priority/P0 Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed. label Jan 16, 2016
@thaJeztah
Copy link
Member

@mavenugo @sanimej any update on this one?

@mavenugo mavenugo mentioned this issue Jan 20, 2016
Merged
@sanimej sanimej mentioned this issue Jan 22, 2016
Merged
@sebi-hgdata
Copy link
Author

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mavenugo mavenugo

Labels
area/networking kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. priority/P0 Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed.
Projects
None yet
Milestone
1.10.0
Development

No branches or pull requests
4 participants
@thaJeztah @mavenugo @sanimej @sebi-hgdata