Docker swarm networking not work with openstack floating ips

[alissonpmedeiros]

I'm trying to create a cluster with floating ips of the openstack, however when I perform the join in the second VM, only the private ip of the VM is recognized in the ingress network. Even so, swarm is able to deploy services on both VMs, however, if I deploy an application (php admin) in VM1 that should query a database in VM2 (my-sql), communication does not occur because in the ingress networks in both VMs has the option peers a floating ip (of the node in which the swarm was started) and private ip (of the VM that held the join). On the other hand, if I use private ip (that are in same network) to create the cluster, everything happens as it should.

Is it possible to work around this problem and specify the floating ip of openstack at the time of doing the docker swarm join?

VM1
internal ip: 192.168.1.11
floating ip: 10.7.229.134

VM2
internal ip: 192.168.1.9
floating ip: 10.7.229.135

Reproduce:
In VM1: docker swarm init --advertise-addr 10.7.229.134
In VM2: docker swarm join --token SWMTKN-1-3ugdbfymlzzu3eocur7leq5lpft861e1xq1ff2mmeuxjyc4x2c-du8acd24p90miebr45sb86yjh 10.7.229.134:2377

This done, when I inspect the network ingress in VM1 and VM2, I get the following:
in VM1: docker network inspect ingress

[
{
"Name": "ingress",
"Id": "s1bcvrldk764c4uqbfvr9pyhg",
"Created": "2018-12-17T16:44:19.76546053Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "c4344c21d964716d87554515a581c9e6da2c83e5bd13eb8f3d2d0ceaa5fd50a0",
"MacAddress": "02:42:0a:ff:00:02",
"IPv4Address": "10.255.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "912eea04196c",
"IP": "10.7.229.134"
},
{
"Name": "b2c37c424792",
"IP": "192.168.1.9"
}
]
}
]

in VM2: docker network inspect ingress

[
{
"Name": "ingress",
"Id": "s1bcvrldk764c4uqbfvr9pyhg",
"Created": "2018-12-17T16:44:27.94519371Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "b8deda1d6f7d7fb20a7ae2ce53ccb30a844bd7a37cfb36ddf76cd6ea78c68432",
"MacAddress": "02:42:0a:ff:00:03",
"IPv4Address": "10.255.0.3/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "912eea04196c",
"IP": "10.7.229.134"
},
{
"Name": "b2c37c424792",
"IP": "192.168.1.9"
}
]
}
]

In VM1:
docker node ls

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
89qbyvpqplno7zv45u5ie4dng * ubuntu1 Ready Active Leader 18.09.0
ilj6ohk1senwlzjunymzy9wya ubuntu2 Ready Active 18.09.0

docker node inspect ubuntu1

[
{
"ID": "89qbyvpqplno7zv45u5ie4dng",
"Version": {
"Index": 9
},
"CreatedAt": "2018-12-17T16:44:18.335495828Z",
"UpdatedAt": "2018-12-17T16:44:19.249108871Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "ubuntu1",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 4000000000,
"MemoryBytes": 2096840704
},
"Engine": {
"EngineVersion": "18.09.0",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUMelG0mPipbJN3CgNLlwW1epTckkwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTgxMjE3MTYzOTAwWhcNMzgxMjEyMTYz\nOTAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABPAB23p1jgjeMCP06tKD/ra1JY6PFvvf0skdOo5JEL6THznKyVOSi0YtO4vx\njy4ye7CETp2uM1kX43h96ULZajujQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBS7lLC7MXGHzrEy48okqFVYWtG0vDAKBggqhkjO\nPQQDAgNJADBGAiEAv4exHtgXK5rrtumoTVc7JLk0XhtLJxDhJeKLrKuL32sCIQDL\nkpH8MF4/EiIv2hw1gTmgDooOBLlbv0vaIGE/pi8GnA==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8AHbenWOCN4wI/Tq0oP+trUljo8W+9/SyR06jkkQvpMfOcrJU5KLRi07i/GPLjJ7sIROna4zWRfjeH3pQtlqOw=="
}
},
"Status": {
"State": "ready",
"Addr": "10.7.229.134"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "10.7.229.134:2377"
}
}
]

docker node inspect ubuntu2

[
{
"ID": "ilj6ohk1senwlzjunymzy9wya",
"Version": {
"Index": 15
},
"CreatedAt": "2018-12-17T16:44:25.224214645Z",
"UpdatedAt": "2018-12-17T16:44:26.242017466Z",
"Spec": {
"Labels": {},
"Role": "worker",
"Availability": "active"
},
"Description": {
"Hostname": "ubuntu2",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 4000000000,
"MemoryBytes": 2096840704
},
"Engine": {
"EngineVersion": "18.09.0",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUMelG0mPipbJN3CgNLlwW1epTckkwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTgxMjE3MTYzOTAwWhcNMzgxMjEyMTYz\nOTAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABPAB23p1jgjeMCP06tKD/ra1JY6PFvvf0skdOo5JEL6THznKyVOSi0YtO4vx\njy4ye7CETp2uM1kX43h96ULZajujQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBS7lLC7MXGHzrEy48okqFVYWtG0vDAKBggqhkjO\nPQQDAgNJADBGAiEAv4exHtgXK5rrtumoTVc7JLk0XhtLJxDhJeKLrKuL32sCIQDL\nkpH8MF4/EiIv2hw1gTmgDooOBLlbv0vaIGE/pi8GnA==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8AHbenWOCN4wI/Tq0oP+trUljo8W+9/SyR06jkkQvpMfOcrJU5KLRi07i/GPLjJ7sIROna4zWRfjeH3pQtlqOw=="
}
},
"Status": {
"State": "ready",
"Addr": "10.7.229.135"
}
}
]

Note that when I inspect VM2, the floating ip is shown in the ManagerStatus part

Cluster visualizer from portainer:

[alissonpmedeiros]

tail -f /var/log/syslog | grep dockerd

Dec 17 17:24:20 ubuntu1 dockerd[1028]: time="2018-12-17T17:24:20.775686695Z" level=info msg="NetworkDB stats ubuntu1(912eea04196c) - netID:s1bcvrldk764c4uqbfvr9pyhg leaving:false netPeers:2 entries:10 Queue qLen:0 netMsg/s:0"