Encrypted overlay uses wrong node address for tunnels

[benkoller]

Description

We're setting up a clean swarm (17.06.2, mix of ubuntu 14.04 and 16.04) between two hosters. To secure traffic for some services we're using an encrypted overlay network. Machines at hoster A can talk to each other over the encrypted overlay, same for hoster B, but traffic between hosters does not flow. sudo ip -s xfrm state on all machines reveals odd tunnels being created.

For the sake of continuity lets assume these conditions to be correct:

• VM1(hosterA): 100.x.x.1 public, 192.168.0.1 private
• VM2(hosterB): 50.x.x.1 public, 172.22.0.1 private
• VM3(hosterB): 50.x.x.2 public, 172.22.0.2 private

VM2 and VM3 have their eth0 set to their local addr., VM1 has its eth0 set to its public ip. This, I think, is the culprit here.

sudo ip -s xfrm state on VM1(hosterA) shows these tunnels:

src 172.22.0.1 >>VM2(hosterB)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0x2b6a593c(728389948) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.1 >>VM2(hosterB)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0x63f3262d(1676879405) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 100.x.x.1 >>VM1(hosterA)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0x63f3262d(1676879405) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.1 >>VM2(hosterB)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0x483521cd(1211441613) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 100.x.x.1 >>VM1(hosterA)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0x10372a49(272050761) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 100.x.x.1 >>VM1(hosterA)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0x047b1c41(75177025) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 100.x.x.1 >>VM1(hosterA)<< dst 100.x.x.1 >>VM1(hosterA)<<
    proto esp spi 0xa2b65be4(2729860068) reqid 13681891(0x00d0c4e3) mode transport
(...)

And vice versa, the same command run on VM2(hosterB):

src 172.22.0.1 >>VM2(hosterB)<< dst 172.22.0.2 >>VM3(hosterB)<<
    proto esp spi 0x3d503b25(1028668197) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.2 >>VM3(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0x9eab976f(2662045551) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.1 >>VM2(hosterB)<< dst  50.x.x.1 >>VM2(hosterB)<<
    proto esp spi 0x01d29cfd(30579965) reqid 13681891(0x00d0c4e3) mode transport
(...)
src  50.x.x.1 >>VM2(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0xde932ff7(3734188023) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.2 >>VM3(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0x72420ef5(1916931829) reqid 13681891(0x00d0c4e3) mode transport
(...)
src 172.22.0.2 >>VM3(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0xb3766530(3010880816) reqid 13681891(0x00d0c4e3) mode transport
(...)
src  50.x.x.1 >>VM2(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0x01d29cfd(30579965) reqid 13681891(0x00d0c4e3) mode transport
(...)
src  50.x.x.1 >>VM2(hosterB)<< dst 172.22.0.1 >>VM2(hosterB)<<
    proto esp spi 0x78df4f38(2027900728) reqid 13681891(0x00d0c4e3) mode transport
(...)

This strikes me as extremely odd, as VM2 and VM3 have been swarm join'ed with --advertise-addr=>>publicIP of VM<<.

This, consecutively, leads to a situation where VM2 and VM3 (because they're at the same hoster and connected via local network, not just public network) can send traffic via the encrypted overlay, but there is no traffic between VM1 and the others.

nc -z -v -u / nc -z -v confirm ports are open.

ESP is allowed via security groups / iptables:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"
DROP       udp  --  anywhere             anywhere             udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"
ACCEPT     udp  --  anywhere             anywhere             policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
DROP       udp  --  anywhere             anywhere             udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     udp  --  anywhere             anywhere             udp spt:isakmp dpt:isakmp
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     ah   --  anywhere             anywhere
ACCEPT     udp  --  anywhere             anywhere             udp spt:7946 dpt:7946
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:7946 dpt:7946
ACCEPT     udp  --  anywhere             anywhere             udp spt:4789 dpt:4789
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2376
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2377

I would have expected docker to create the ipsec tunnels not with the address stored for eth0 but for the address given as --advertise-addr=.

Is there a way to force docker to follow my expected behaviour / correct the behaviour during swarm join?

Steps to reproduce the issue:

1. Have some nodes with eth0 set to local address, some with eth0 set to public addr.
2. Join all nodes to swarm successfully
3. docker stack deploy a stack with encrypted overlay network to swarm

Describe the results you received:

ipsec tunnels created by docker are not public-ip to public-ip but local-ip to public-ip and therefore don't work

Describe the results you expected:

ipsec tunnels created by docker are created from --advertise-addr= to --adveritise-addr=

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

(paste your output here)

Output of docker info:

Same on all VMs

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 20:00:33 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:26 2017
 OS/Arch:      linux/amd64
 Experimental: false

Additional environment details (AWS, VirtualBox, physical, etc.):

If it helps:

• VM1 is at a oldschool hoster, no local networking, only a public IP address for the box
• VM2 and VM3 are at MS Azure

[thaJeztah]

ping @mavenugo @fcrisciani PTAL