Cannot ping between containers via overlay net_diffhost

[wpecker]

Description of problem:
I setup 1 kvstore and 2 engines on Debian 8, node-1 and node-2.
Then starting a etcd instance in kvstore.

docker version (on node-1:192.168.1.2 and node-2:192.168.2.5 )
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:39 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:39 2016
OS/Arch: linux/amd64

node-1:192.168.1.2
/usr/bin/docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-advertise=192.168.1.2:2375 --cluster-store=etcd://192.168.2.5:2379/swarm_dgateway

node-2:192.168.2.5
/usr/bin/docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-advertise=192.168.2.5:2375 --cluster-store=etcd://192.168.2.5:2379/swarm_dgateway

etcd instance docker-compose.yml as below:
etcd:
image: quay.io/coreos/etcd:latest
restart: always
ports:
- "2379:2379"
- "2380:2380"
- "4001:4001"
volumes:
- /usr/share/ca-certificates/:/etc/ssl/certs
command: >
-name etcd0
-advertise-client-urls http://192.168.2.5:2379,http://192.168.2.5:4001
-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
-initial-advertise-peer-urls http://192.168.2.5:2380
-listen-peer-urls http://0.0.0.0:2380
-initial-cluster-token etcd-cluster-1
-initial-cluster etcd0=http://192.168.2.5:2380
-initial-cluster-state new

check etcd health
$ docker exec dockerapp_etcd_1 /etcdctl cluster-health
member aafa1c66c1d89ef2 is healthy: got healthy result from http://192.168.2.5:2379
cluster is healthy

node-1:192.168.1.2
$ sudo docker run -itd --name c1_on_diffhost --net net_diffhost ubuntu:14.04
$ sudo docker exec c1_on_diffhost ifconfig

eth0 Link encap:Ethernet HWaddr 02:42:0a:00:02:02
inet addr:10.0.2.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:aff:fe00:202/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:788 (788.0 B) TX bytes:648 (648.0 B)

node-2:192.168.2.5
$ sudo docker network create -d overlay net_diffhost
$ sudo docker run -itd --name c2_on_diffhost --net net_diffhost ubuntu:14.04
$ sudo docker exec c2_on_diffhost ifconfig

eth0 Link encap:Ethernet HWaddr 02:42:0a:00:02:03
inet addr:10.0.2.3 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:aff:fe00:203/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:13 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1038 (1.0 KB) TX bytes:648 (648.0 B)

$ sudo docker exec c2_on_diffhost ping -c 4 10.0.2.2
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.

--- 10.0.2.2 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3027ms

[cpuguy83]

This would seem to indicate that the overlay is not working.
Can you check the daemon logs on both nodes?

Also note the ports used for establishing the overlay:

Protocol	Port	Description
udp	4789	Data plane (VXLAN)
tcp/udp	7946	Control plane

[wpecker]

@cpuguy83
ON node-2:192.168.2.5
$ netstat -na | grep 7946
tcp 0 0 192.168.2.5:7946 0.0.0.0:* LISTEN
tcp 0 0 192.168.2.5:7946 192.168.2.1:63808 TIME_WAIT
tcp 0 0 10.0.4.15:58772 192.168.1.2:7946 TIME_WAIT
tcp 0 0 192.168.2.5:7946 192.168.2.1:63823 TIME_WAIT
udp 0 0 192.168.2.5:7946 0.0.0.0:*

$ netstat -na | grep 4789
udp 0 0 0.0.0.0:4789 0.0.0.0:*

ON node-1:192.168.1.2
tcp 0 0 192.168.1.2:7946 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.2:7946 192.168.1.3:49003 TIME_WAIT
tcp 0 0 192.168.1.2:7946 192.168.1.1:63978 TIME_WAIT
tcp 0 0 192.168.1.2:7946 192.168.1.4:37270 TIME_WAIT
udp 0 0 192.168.1.2:7946 0.0.0.0:*

udp 0 0 0.0.0.0:4789 0.0.0.0:*

[wpecker]

@cpuguy83
check both node, the docker daemon log, nothing return when exec: sudo docker exec c2_on_diffhost ping -c 4 10.0.2.2

[michaljrk]

+1

I've faced very similar situation (the overlay network was working but stopped at some point , probably after one of the server restart). Now the situation looks pretty much the same as swan-go described.

Environment details:

docker version
Client:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

etcd

3-node cluster

etcd Version: 2.2.5

~# etcdctl --endpoint "http://xxx" cluster-health
member 382948f65a4292d0 is healthy: got healthy result from http://xxx
member c6a23a34da0740b9 is healthy: got healthy result from http://yyy
member dbdc512eab45cf29 is healthy: got healthy result from http://zzz
cluster is healthy

iptables rules

-A INPUT -s xxx/22 -p tcp -m tcp --dport 7946 -j ACCEPT
-A INPUT -s xxx/22 -p udp -m udp --dport 4789 -j ACCEPT
-A INPUT -s xxx/22 -p udp -m udp --dport 7946 -j ACCEPT

I had to create another overlay network and connect containers to it to recover connectivity.

[wpecker]

:) This issue can't sink

[realcbb]

Does anyone know what the problem is?

[wpecker]

This issue have been sink, I don't know the newest version about this issue.

[thaJeztah]

overlay with external kvstores was deprecated and removed.
Let me close this ticket for now, as it looks like it went stale.