Container cannot access another containers in one overlay network on same node

[BSWANG]

Sometimes in one overlay network and only on one node. The container cannot access another containers in one overlay network. But I don't know how to reproduce this issue.

I use the tcpdump to catch the packages in overlay network namespace. Found the packages not be forward to other container's veth. But be forward to vxlan0.

tcpdump at connection issue:

/ # tcpdump -i vxlan0 -vv -nn icmp
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:48:18.750989 IP (tos 0x0, ttl 64, id 29257, offset 0, flags [DF], proto ICMP (1), length 84)
    172.19.0.2 > 172.19.0.4: ICMP echo request, id 33536, seq 0, length 64

tcpdump at normal:

/# tcpdump -i vxlan0 -vv -nn icmp
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Check the br0 forward table. The br0 bridge mac table's port is different with normal state:

/ # brctl showmacs br0
port no	mac addr		is local?	ageing timer
  1	02:42:ac:13:00:02	no		   0.22
  1	02:42:ac:13:00:04	no		  11.51
  1	02:42:ac:13:00:05	no		   8.87
  1	02:42:ac:13:00:06	no		   3.21
  1	02:42:ac:13:00:07	no		   9.22
  1	02:42:ac:13:00:08	no		   3.86
  1	02:42:ac:13:00:09	no		   2.50
  3	0a:9c:03:9c:2e:3e	yes		   0.00
  7	0a:e7:50:45:5d:85	yes		   0.00
  6	32:b9:28:74:46:2e	yes		   0.00
  9	6e:98:31:01:26:a4	yes		   0.00
  5	76:40:c5:a5:18:ca	yes		   0.00
  2	86:22:54:09:f2:05	yes		   0.00
  4	c2:16:7c:9a:5c:a8	yes		   0.00
  1	c6:08:88:0b:da:3c	yes		   0.00
  8	f2:35:65:d2:73:3a	yes		   0.00
/ # bridge fdb
0a:9c:03:9c:2e:3e dev veth1 master br0 permanent
02:42:ac:13:00:08 dev vxlan0 master br0
f2:35:65:d2:73:3a dev veth7 master br0 permanent
02:42:ac:13:00:02 dev vxlan0 master br0
c2:16:7c:9a:5c:a8 dev veth13 master br0 permanent
86:22:54:09:f2:05 dev veth0 master br0 permanent
c6:08:88:0b:da:3c dev vxlan0 master br0 permanent
02:42:ac:13:00:05 dev vxlan0 master br0
6e:98:31:01:26:a4 dev veth10 master br0 permanent
0a:e7:50:45:5d:85 dev veth6 master br0 permanent
76:40:c5:a5:18:ca dev veth18 master br0 permanent
32:b9:28:74:46:2e dev veth11 master br0 permanent
02:42:ac:13:00:04 dev vxlan0 master br0
02:42:ac:13:00:09 dev vxlan0 master br0
02:42:ac:13:00:06 dev vxlan0 master br0
02:42:ac:13:00:07 dev vxlan0 master br0
02:42:ac:13:00:02 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:04 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:05 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:06 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:07 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:08 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:09 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
33:33:00:00:00:01 dev veth0 self permanent
01:00:5e:00:00:01 dev veth0 self permanent
33:33:00:00:00:01 dev veth1 self permanent
01:00:5e:00:00:01 dev veth1 self permanent
33:33:00:00:00:01 dev veth6 self permanent
01:00:5e:00:00:01 dev veth6 self permanent
33:33:00:00:00:01 dev veth7 self permanent
01:00:5e:00:00:01 dev veth7 self permanent
33:33:00:00:00:01 dev veth10 self permanent
01:00:5e:00:00:01 dev veth10 self permanent
33:33:00:00:00:01 dev veth11 self permanent
01:00:5e:00:00:01 dev veth11 self permanent
33:33:00:00:00:01 dev veth13 self permanent
01:00:5e:00:00:01 dev veth13 self permanent
33:33:00:00:00:01 dev veth18 self permanent
01:00:5e:00:00:01 dev veth18 self permanent

brctl showmacs br0 at normal state:

/ # brctl showmacs br0
port no	mac addr		is local?	ageing timer
  2	02:42:ac:13:00:02	no		   0.32
  4	02:42:ac:13:00:04	no		   0.32
  5	02:42:ac:13:00:05	no		   0.79
  6	02:42:ac:13:00:06	no		   0.39
  7	02:42:ac:13:00:07	no		   0.39
  8	02:42:ac:13:00:08	no		   0.79
  9	02:42:ac:13:00:09	no		   0.50
  3	0a:9c:03:9c:2e:3e	yes		   0.00
  7	0a:e7:50:45:5d:85	yes		   0.00
  6	32:b9:28:74:46:2e	yes		   0.00
  9	6e:98:31:01:26:a4	yes		   0.00
  5	76:40:c5:a5:18:ca	yes		   0.00
  2	86:22:54:09:f2:05	yes		   0.00
  4	c2:16:7c:9a:5c:a8	yes		   0.00
  1	c6:08:88:0b:da:3c	yes		   0.00
  8	f2:35:65:d2:73:3a	yes		   0.00

docker version and system info

# lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.2.1511 (Core)
Release:	7.2.1511
Codename:	Core

# uname -a
Linux c81509ccbcb5d4d76bcdf2267f3fc3556-node1 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# docker version
Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   a176465
 Built:        Wed Sep 20 02:03:17 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   a176465
 Built:        Wed Sep 20 02:04:53 2017
 OS/Arch:      linux/amd64
 Experimental: false

The workaround way I found:

1. recreate br0 by recreate network or restart docker daemon.
2. ip link set br0 down && ip link set br0 up

[sylvainmouquet]

I am wondering if it's the same error than moby/moby#35230

[fcrisciani]

@BSWANG this should had been fix in 17.10, there was a bug that was causing miss configuration in the fdb entries in the overlay bridge causing some of them to not be reachable. Closing this for now, let me know if you still see this issue on 17.10