Docker swarm network container connection timeout

[sylvainmouquet]

Hello,

I use docker swarm with 3 nodes (docker version 17.09.0-ce)
The services are attached to one overlay network

                    "Subnet": "172.28.0.0/16",
                    "IPRange": "172.28.5.0/24",
                    "Gateway": "172.28.5.254"

A container has ip 172.28.5.25 in manager 3
When i try to connect to this container in manager 3 it works
When i try to connect to this container in manager 2 it fails
When i try to connect to this container in manager 1 it works

Manager1

$ docker exec -it 9b bash
root@9bfc123c9595:/# netcat -z -v -w 3 172.28.5.25 85 2>&1
xyz_service.1.s4ylfly3dq7em3proz5w7ps5i.xyz_network [172.28.5.25] 85 (?) open

Manager2

$ docker exec -it c3c bash
root@c3c32994e86e:/# netcat -z -v -w 3 172.28.5.25 85 2>&1
xyz_service.1.s4ylfly3dq7em3proz5w7ps5i.xyz_network [172.28.5.25] 85 (?) : Connection timed out

Manager3

$ docker exec -it a5429f43f012 bash
root@a5429f43f012:/brazil/server/node# netcat -z -v -w 3 172.28.5.25 85 2>&1
2915bf9b49d0.xyz_network [172.28.5.25] 85 (?) open

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.18.0.1      0.0.0.0         UG    0      0        0 eth1
10.255.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth2
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1
172.28.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0

Why the gateway is not 172.28.5.254 ?
Why i have this timeout ?

@fcrisciani

[sylvainmouquet]

Next i kill the container but i get the same IP and the connection fails yet.
I update the service, i have 2 containers and now with the new container and a new IP the connection is a success

[fcrisciani]

@sylvainmouquet
you have 3 interfaces in the container:

• eth2 is the ingress network
• eth1 is the default gw network
• eth0 is your overlay network

I think can be an issue with the overlay fdb entry, if you are able to reproduce the issue, follow these steps here to verify if the fdb is miss configured: #32195 (comment)