ingress network(s) not shared across swarm #25386

Closed
Garreat opened this Issue Aug 4, 2016 · 21 comments

Projects

None yet

8 participants

@Garreat
Garreat commented Aug 4, 2016 edited

Output of docker version:

# docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.12.0
Storage Driver: devicemapper
 Pool Name: docker-253:1-25533201-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 765.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 4.896 GB
 Metadata Space Used: 1.516 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay null host
Swarm: active
 NodeID: 3w1uafd49ulophb6hqb5rwzne
 Is Manager: true
 ClusterID: 5ogz1qyltdgd7y8k5ulgoixxl
 Managers: 2
 Nodes: 6
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: 10.48.106.138
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.3 MiB
Name: plkrdockmasterq2
ID: LOM6:YS7J:27DA:WENI:CKEQ:GP2Z:6HX4:D5NS:IZF2:WR25:ZWXJ:VPBI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 10.48.106.151:5000
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
VMware, physical
All machines in same VLAN (all traffic open); OS firewalls disabled

Steps to reproduce the issue:

  1. docker swarm init
  2. docker swarm join...
  3. (now there is a swarm set up, let's run a service)
  4. docker service create --replicas 2 --name app --network ingress...
  5. docker network inspect ingress

Describe the results you received:
on node 1:

q1 ~]# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "elvyrujrkowm01sc2t50p5lyd",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "207e00880eead9491a7942e001b38f77489b11966e60df0694ab3e0f0cbce85c": {
                "Name": "app.1.eqzsim28dhl1ffzorf5wvphgk",
                "EndpointID": "a9be20d7afdb4491a30c9b9f084671470493f56f3decd36d285febf4ae48293a",
                "MacAddress": "02:42:0a:ff:00:0c",
                "IPv4Address": "10.255.0.12/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "704dbd527c912f7d3dce15473012c2b4555b35d72123a6824525722b28708727",
                "MacAddress": "02:42:0a:ff:00:03",
                "IPv4Address": "10.255.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "256"
        },
        "Labels": {}
    }
]

on node 2:

q2 ~]# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "elvyrujrkowm01sc2t50p5lyd",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "ed0d54e44ce5065dfb0e1d4493f3743ee04bc5b30b01100fb2790ea5612c5921": {
                "Name": "app.2.49j3avqu207jnt9n9qkq512o8",
                "EndpointID": "93c776f6be41ab492a4eb48aae59a47fe8cc8dccc15f411513ab6a62570c66ae",
                "MacAddress": "02:42:0a:ff:00:0d",
                "IPv4Address": "10.255.0.13/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "b17ea877bd3d890e9a9406ae21cb80357b45ec02bfa02b19e055064a537b1d8e",
                "MacAddress": "02:42:0a:ff:00:04",
                "IPv4Address": "10.255.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "256"
        },
        "Labels": {}
    }
]

Describe the results you expected:
Inspecting the network should show all containers attached.
Meanwhile, I can only see the task containers running on the node where docker network inspect ingress was ran.
Effectively, my whole setup makes no sense. 502 bad gateway.

Additional information you deem important (e.g. issue happens only occasionally):
This was ran inside app.2:

[root@ed0d54e44ce5 /]# ping 10.255.0.12
PING 10.255.0.12 (10.255.0.12) 56(84) bytes of data.
64 bytes from 10.255.0.12: icmp_seq=1 ttl=64 time=0.596 ms
64 bytes from 10.255.0.12: icmp_seq=2 ttl=64 time=0.473 ms
64 bytes from 10.255.0.12: icmp_seq=3 ttl=64 time=0.514 ms
64 bytes from 10.255.0.12: icmp_seq=4 ttl=64 time=0.421 ms
64 bytes from 10.255.0.12: icmp_seq=5 ttl=64 time=0.383 ms
^C
--- 10.255.0.12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.383/0.477/0.596/0.076 ms
[root@ed0d54e44ce5 /]# ping app.1           
ping: unknown host app.1
[root@ed0d54e44ce5 /]# ping app.2           
ping: unknown host app.2
[root@ed0d54e44ce5 /]# cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0

I have created another overlay network on master node, but only second master node could see it in docker network ls output (?).
All nodes are in same VLAN (all traffic open); OS firewalls disabled

I had no issue running this entire multi-host overlay stack pre-1.12.
Suggestions?

@tectoro
tectoro commented Aug 4, 2016

Same issue here. We are currently running all our multi-host environments on docker 1.11 using consul. Works perfectly.

We created a new environment in digital ocean with 5 hosts. Installed docker 1.12. Made first node as manager. Joined other nodes. Creating services and sharing them across hosts all works fine.

Once we update one of the service to change a param, it starts tearing apart. DNS resolution of existing services fails. Load balancing of existing services starts misbehaving.

Used --listen-addr and --adertise-addr flags while joining the nodes into cluster.

@justincormack
Member

You should not be directly using the ingress network, that is for routing
internal traffic. Can you create a new overlay network and connect your
containers to that and try again?

On 4 Aug 2016 2:49 a.m., "Jacek" notifications@github.com wrote:

Output of docker version:

docker version

Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64

Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64

Output of docker info:

docker info

Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:1-25533201-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 765.9 MB
Data Space Total: 107.4 GB
Data Space Available: 4.896 GB
Metadata Space Used: 1.516 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay null host
Swarm: active
NodeID: 3w1uafd49ulophb6hqb5rwzne
Is Manager: true
ClusterID: 5ogz1qyltdgd7y8k5ulgoixxl
Managers: 2
Nodes: 6
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot interval: 10000
Heartbeat tick: 1
Election tick: 3
Dispatcher:
Heartbeat period: 5 seconds
CA configuration:
Expiry duration: 3 months
Node Address: 10.48.106.138
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.3 MiB
Name: plkrdockmasterq2
ID: LOM6:YS7J:27DA:WENI:CKEQ:GP2Z:6HX4:D5NS:IZF2:WR25:ZWXJ:VPBI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
10.48.106.151:5000
127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
VMware, physical
All machines in same VLAN (all traffic open); OS firewalls disabled

Steps to reproduce the issue:

  1. docker swarm init
  2. docker swarm join...
  3. (now there is a swarm set up, let's run a service)
  4. docker service create --replicas 2 --name app --network ingress...
  5. docker network inspect ingress

Describe the results you received:
on node 1:

q1 ~]# docker network inspect ingress
[
{
"Name": "ingress",
"Id": "elvyrujrkowm01sc2t50p5lyd",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Containers": {
"207e00880eead9491a7942e001b38f77489b11966e60df0694ab3e0f0cbce85c": {
"Name": "app.1.eqzsim28dhl1ffzorf5wvphgk",
"EndpointID": "a9be20d7afdb4491a30c9b9f084671470493f56f3decd36d285febf4ae48293a",
"MacAddress": "02:42:0a:ff:00:0c",
"IPv4Address": "10.255.0.12/16",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "704dbd527c912f7d3dce15473012c2b4555b35d72123a6824525722b28708727",
"MacAddress": "02:42:0a:ff:00:03",
"IPv4Address": "10.255.0.3/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "256"
},
"Labels": {}
}
]

on node 2:

q2 ~]# docker network inspect ingress
[
{
"Name": "ingress",
"Id": "elvyrujrkowm01sc2t50p5lyd",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Containers": {
"ed0d54e44ce5065dfb0e1d4493f3743ee04bc5b30b01100fb2790ea5612c5921": {
"Name": "app.2.49j3avqu207jnt9n9qkq512o8",
"EndpointID": "93c776f6be41ab492a4eb48aae59a47fe8cc8dccc15f411513ab6a62570c66ae",
"MacAddress": "02:42:0a:ff:00:0d",
"IPv4Address": "10.255.0.13/16",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "b17ea877bd3d890e9a9406ae21cb80357b45ec02bfa02b19e055064a537b1d8e",
"MacAddress": "02:42:0a:ff:00:04",
"IPv4Address": "10.255.0.4/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "256"
},
"Labels": {}
}
]

Describe the results you expected:
Inspecting the network should show all containers attached.
Meanwhile, I can only see the task containers running on the node where docker
network inspect ingress was ran.
Effectively, my whole setup makes no sense. 502 bad gateway.

Additional information you deem important (e.g. issue happens only
occasionally):

This was ran inside app.2:

[root@ed0d54e /]# 10.255.0.12
PING 10.255.0.12 (10.255.0.12) 56(84) bytes of data.
64 bytes from 10.255.0.12: icmp_seq=1 ttl=64 time=0.596 ms
64 bytes from 10.255.0.12: icmp_seq=2 ttl=64 time=0.473 ms
64 bytes from 10.255.0.12: icmp_seq=3 ttl=64 time=0.514 ms
64 bytes from 10.255.0.12: icmp_seq=4 ttl=64 time=0.421 ms
64 bytes from 10.255.0.12: icmp_seq=5 ttl=64 time=0.383 ms
^C
--- 10.255.0.12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.383/0.477/0.596/0.076 ms
[root@ed0d54e /]# ping app.1
ping: unknown host app.1
[root@ed0d54e /]# ping app.2
ping: unknown host app.2
[root@ed0d54e /]# cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

I have created another overlay network on master node, but only second
master node could see it in docker network ls output (?).
All nodes are in same VLAN (all traffic open); OS firewalls disabled

I had no issue running this entire multi-host overlay stack pre-1.12.
Suggestions?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#25386, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPGxrOcLRc829ozdrJz3WUwptXUQKks5qcTcygaJpZM4JcPKr
.

@Garreat
Garreat commented Aug 4, 2016 edited

@tectoro
I have rebuilt the cluster using --listen-addr and --advertise-addr for all nodes (managers and workers). No effect.

@justincormack
As mentioned in original post: I have created another overlay network on master node, but only second master node could see it in docker network ls output (?).
For worker nodes, the freshly created overlay network is invisible. Just re-tried that on freshly rebuilt cluster, with only 1 manager node. No effect.

There are no firewalls, time is NTP-synced. Looking at the ammount of similar issues, wasn't 1.12 GA a bit rushed eh...?
Thank you -- all input is very welcome.

EDIT
For the record, these are my actual outputs:
docker node ls

[root@plkrdockmasterq1 ~]# docker node ls
ID                           HOSTNAME          STATUS  AVAILABILITY  MANAGER STATUS
2xzkc4pgj83eb8ah9rxc8ec0r    plkrdockfrontq1   Ready   Active        
4lu2g1g3nhe1f685buwqst46n    plkrdocklbq1      Ready   Active        
9runs068bcny66c6y9h97uc52    plkrdockfrontq2   Ready   Active        
aslxopjund18tyctesw951zmx    plkrdocknq1       Ready   Active        
cripwcmi8uaazgjhistu0dqxn *  plkrdockmasterq1  Ready   Active        Leader

systemctl status docker -l @ leader

Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00.371993794+02:00" level=info msg="raft.node: 72c412e5aee2a6d4 elected leader 72c412e5aee2a6d4 at term 2"
Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00.702105731+02:00" level=info msg="Initializing Libnetwork Agent Local-addr=10.48.106.137 Adv-addr=10.48.106.137 Remote-addr ="
Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00.702207191+02:00" level=info msg="Initializing Libnetwork Agent Local-addr=10.48.106.137 Adv-addr=10.48.106.137 Remote-addr ="
Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00.709044744+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00.709082883+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Aug 04 10:48:00 plkrdockmasterq1 dockerd[17415]: time="2016-08-04T10:48:00+02:00" level=info msg="Firewalld running: false"

systemctl status docker -l @ worker

Aug 04 10:48:39 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T10:48:39.360950531+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Aug 04 10:48:39 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T10:48:39.360981225+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Aug 04 10:48:39 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T10:48:39+02:00" level=info msg="Firewalld running: false"
Aug 04 12:00:39 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T12:00:39.355237479+02:00" level=error msg="Bulk sync to node plkrdockfrontq2 timed out"
Aug 04 12:03:09 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T12:03:09.354813633+02:00" level=error msg="Bulk sync to node plkrdockfrontq2 timed out"
Aug 04 12:03:39 plkrdockfrontq1 dockerd[19641]: time="2016-08-04T12:03:39.355896622+02:00" level=error msg="Bulk sync to node plkrdockfrontq2 timed out"
@justincormack
Member

Networks are only visible on nodes when a task connected to that network is
running on that node. Starting a service will still work.

On 4 Aug 2016 10:52 a.m., "Jacek" notifications@github.com wrote:

@tectoro https://github.com/tectoro
I have rebuilt the cluster using --listen-addr and --advertise-addr for
all nodes (managers and workers). No effect.

@justincormack https://github.com/justincormack
As mentioned in original post: I have created another overlay network on
master node, but only second master node could see it in docker network ls
output (?).
For worker nodes, the freshly created overlay network is invisible. Just
re-tried that on freshly rebuilt cluster.

There are no firewalls, time is NTP-synced. Looking at the ammount of
similar issues, wasn't 1.12 GA a bit rushed eh...?
Thank you -- all input is very welcome.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#25386 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPFX6rl7XUQmCKSKRokVg-GPVbm8Aks5qcahAgaJpZM4JcPKr
.

@rogaha
Contributor
rogaha commented Aug 4, 2016 edited

@justincormack I've raised this issue (#25396) it will hopefully avoids this issue #25386 (comment) in the future.

@Garreat
Garreat commented Aug 4, 2016

@justincormack
This is what happens when I try to start a service attached to overlay network blah:

[root@plkrdockmasterq1 1.12]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
djkuhim07w9w        blah                overlay             swarm               
dff86ceae892        bridge              bridge              local               
427780acbefc        docker_gwbridge     bridge              local               
3a5cdae67716        host                host                local               
bsyevfhypohr        ingress             overlay             swarm               
49ae7c9f9dd3        none                null                local
[root@plkrdockfrontq1 ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
606a04e50193        bridge              bridge              local               
e7756ec3169f        docker_gwbridge     bridge              local               
a11a0e707bc8        host                host                local               
bsyevfhypohr        ingress             overlay             swarm               
e0eadf1c0c0c        none                null                local

Let's proceed:

docker service create \
    --replicas 2 \
    --name www \
    --update-delay 10s \
    --update-parallelism 1 \
    --restart-condition none \
    --constraint 'node.labels.role == front' \
    --network blah \    
    10.48.106.151:5000/nginx-worker

It's indeed running docker service ps www:

[root@plkrdockmasterq1 1.12]# docker service ps www
ID                         NAME   IMAGE                            NODE             DESIRED STATE  CURRENT STATE          ERROR
117f5y8f1wymeo1ovzyy8hpm5  www.1  10.48.106.151:5000/nginx-worker  plkrdockfrontq2  Running        Running 5 minutes ago  
1fkcgwcxde8nv4ju97lg9ujv8  www.2  10.48.106.151:5000/nginx-worker  plkrdockfrontq1  Running        Running 5 minutes ago

Now I can see the network @ worker, but only with worker's own container docker network inspect blah:

[root@plkrdockfrontq1 ~]# docker network inspect blah
[
    {
        "Name": "blah",
        "Id": "djkuhim07w9wnz3emjyq2yrli",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "8f7ae03ae9330a556bbc1cb82368bc451a2f2e952662271afadd66c2b40d812f": {
                "Name": "www.2.1fkcgwcxde8nv4ju97lg9ujv8",
                "EndpointID": "60828dc2e5bb6661bd27fd3f80b7e2950de23101bc577172896040d88a2da5cb",
                "MacAddress": "02:42:0a:00:00:07",
                "IPv4Address": "10.0.0.7/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]
[root@plkrdockfrontq2 ~]# docker network inspect blah
[
    {
        "Name": "blah",
        "Id": "djkuhim07w9wnz3emjyq2yrli",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "8867651feea8fe64d1acc32aed9bb863fff98f071d655b6a9e9f34795a64d23e": {
                "Name": "www.1.117f5y8f1wymeo1ovzyy8hpm5",
                "EndpointID": "cec7b6fac0baed1763b4b214237463c372259f426dc4eaa3dd31fa46c251fd65",
                "MacAddress": "02:42:0a:00:00:06",
                "IPv4Address": "10.0.0.6/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

And again, from inside the container:

[root@8f7ae03ae933 /]# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from 10.0.0.7: icmp_seq=2 ttl=64 time=0.064 ms
^C
--- 10.0.0.7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.064/0.076/0.089/0.015 ms
[root@8f7ae03ae933 /]# ping 10.0.0.6
PING 10.0.0.6 (10.0.0.6) 56(84) bytes of data.
64 bytes from 10.0.0.6: icmp_seq=1 ttl=64 time=1.77 ms
64 bytes from 10.0.0.6: icmp_seq=2 ttl=64 time=0.559 ms
64 bytes from 10.0.0.6: icmp_seq=3 ttl=64 time=0.607 ms
64 bytes from 10.0.0.6: icmp_seq=4 ttl=64 time=0.498 ms
64 bytes from 10.0.0.6: icmp_seq=5 ttl=64 time=0.488 ms
64 bytes from 10.0.0.6: icmp_seq=6 ttl=64 time=0.535 ms
^C
--- 10.0.0.6 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5002ms
rtt min/avg/max/mdev = 0.488/0.744/1.777/0.463 ms
[root@8f7ae03ae933 /]# ping www.1   
ping: unknown host www.1
[root@8f7ae03ae933 /]# ping www.2
ping: unknown host www.2
[root@8f7ae03ae933 /]#

So we made a circle here - you were perfectly right with the overlay network visibility tho.

@rogaha We have already drifted away from the ingress network

@tectoro
tectoro commented Aug 4, 2016 edited

@justincormack Actually I started the whole setup with a overlay network. Since it did not work as expected, I tried with the default "ingress" network.

Also, I did the whole switch off firewall, NTP sync thing.. No luck.

@Garreat
Garreat commented Aug 4, 2016 edited

Cruising through issues here at github, I have encountered #24996 .
I have noticed that not all of my nodes have ip_vs module loaded. So I did modprobe ip_vs load it everywhere, systemctl docker restart + cluster rebuild. No luck.

Nevertheless, I have started the full stack within blah overlay:

ID            NAME  REPLICAS  IMAGE                                  COMMAND
2thxegt4sh3g  www   2/2       10.48.106.151:5000/nginx-worker        
7285v7xha7a9  lb    1/1       10.48.106.151:5000/nginx-lb            
9ahsk5n36695  app   2/2       10.48.106.151:5000/uwsgi-flask-docker  uwsgi --ini /app/app.ini

Then, I have checked the logs of lb load balancer task (http 502 forever). Some interesting stuff there:

2016/08/04 13:52:33 [error] 7#0: *1 no live upstreams while connecting to upstream, client: 10.255.0.9, server: my.site, request: "GET / HTTP/1.1", upstream: "uwsgi://apps", host: "10.48.106.134"

What is that 10.255.0.9 address...? The load balancer is actually...

"Containers": {
            "4dacc7805b4ae4336b9f9f945d5d121244da287e27b29f3c988054224460e934": {
                "Name": "lb.1.bbwsems3jmo1dyhrbwe7dexup",
                "EndpointID": "5c7925c8a382ba68baa569d7d3f4ecff7dacac442f907f202e19a3ce719dbd66",
                "MacAddress": "02:42:0a:00:00:09",
                "IPv4Address": "10.0.0.9/24",
                "IPv6Address": ""
            }
        }

This might be the right track - see some error stuff from another node systemctl status docker -l:

Aug 04 15:51:16 plkrdockfrontq1 dockerd[2767]: peerdbupdate in sandbox failed for ip 10.255.0.9 and mac 02:42:0a:ff:00:09: couldn't find the subnet "10.255.0.9/16" in network "aan34tq9652pu2b0q5tci35h1"

Well, hard to find such 10.255.0.9/16 subnet in my blah network - why look for it?

"Name": "blah",
        "Id": "aan34tq9652pu2b0q5tci35h1",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]

Maybe I have drifted too far (lb is published with this fancy new thing I would kindly skip entirely).
I don't know.
Next, I will try to specify my overlay net address/mask manually.

EDIT
Manually specified subnet/range for blah. No success.

@mrjana
Contributor
mrjana commented Aug 4, 2016

@Garreat www.1 is not a valid container name. If you checkout docker ps for these containers (which are part of a service) the name of the container is of the form <service_name>.<slot>.<task_id>. You have to use that name in it's entirety to resolve the name. I am confused with this thread as to what is the issue. But if that is the issue, then it is not an issue. Just the container names are different for services.

@tectoro
tectoro commented Aug 4, 2016

@mrjana Just want to add here.. I have been using just the service name e.g. "mongo" which seems to be not working after a couple of service updates in the cluster. It works absolutely fine when we create all the services afresh. Once we start updating some of the services, they are either not able to resolve the service name or not able to reach the service port.

Also, I have observed that the service to container mapping gets corrupted sometimes where in when I hit a service I see some other service processing that request.

@mrjana
Contributor
mrjana commented Aug 4, 2016

@tectoro If you are seeing issues after a service update you are most like hitting #24789 or another any number of issues created in GH which is a variant of that. This is already fixed here docker/libnetwork#1370 and it will make it's way to docker in a 1.12.1 bug fix release.

@Garreat
Garreat commented Aug 4, 2016 edited

I think you are right @mrjana .
Are we bound to use internal load balancing then? Like @tectoro 's "mongo"?

This lb of mine is nginx gateway to the outside world (https). It is obviously a part of current Swarm setup.
In my hardware setup, I have a certain number of hardware failure domains. It is directly represented in lb's nginx.conf. With this variable task ID:

  • I can expose lb's upstreams as published and use external load balancer
  • include Consul into equation (hoped to quit it :) )

How would you relate to this https://medium.com/@lherrera/poor-mans-load-balancing-with-docker-2be014983e5#.epn5cwcd6 ?
Can I quit nginx for built-in load balancer?

Thank you, now it's mostly clear!

[root@8c36e3fa3888 /]# ping  www.2.3l3088pgcexvp7kcz7whpqjef
PING www.2.3l3088pgcexvp7kcz7whpqjef (100.0.0.7) 56(84) bytes of data.
64 bytes from www.2.3l3088pgcexvp7kcz7whpqjef.blah (100.0.0.7): icmp_seq=1 ttl=64 time=0.626 ms
64 bytes from www.2.3l3088pgcexvp7kcz7whpqjef.blah (100.0.0.7): icmp_seq=2 ttl=64 time=0.456 ms
64 bytes from www.2.3l3088pgcexvp7kcz7whpqjef.blah (100.0.0.7): icmp_seq=3 ttl=64 time=0.406 ms
64 bytes from www.2.3l3088pgcexvp7kcz7whpqjef.blah (100.0.0.7): icmp_seq=4 ttl=64 time=0.462 ms
@mrjana
Contributor
mrjana commented Aug 4, 2016

@Garreat If you want to terminate https in your LB you need nginx. If you can terminate them in your app you directly use the built-in load balancer. But even using nginx as an LB is pretty easy. You can create nginx as a service with ports exposed. In addition to that you can connect nginx service and your app service in the same network. Your app service can be run in default vip mode or dnsrr mode. If you run your app service in dnsrr then when your query the app service name you will get the IPs of all the containers part of the service. You can populate nginx using that information. If you run your app service in vip mode, then you can configure your nginx to just point to the service name as the backend and vip will automatically do the load balancing.

@Garreat
Garreat commented Aug 4, 2016 edited

That helps a lot, thank you!
I definitely need the VIP mode for my upstreams then (app and worker).
NGINX does it's upstream servers resolving on start-time -- unless it is a commercial subscription -- not my case. There is this module https://github.com/GUI/nginx-upstream-dynamic-servers but I've had trouble with it.

Which mode is the default?

Awesome stuff, thank you so much @mrjana . 5star

@mrjana
Contributor
mrjana commented Aug 4, 2016

@Garreat vip mode is the default.

BTW, I am going to close this issue. Please reopen it if you don't think this should be closed. We can still continue the conversation here even if the issue is closed.

@mrjana mrjana closed this Aug 4, 2016
@Garreat
Garreat commented Aug 5, 2016

Ok, with this knowledge I got my app running pretty nicely.

I have seen this statement that eventually docker service shall possess capabilities present in Docker Compose.
Anyway, there's a couple of (topic unrelated) questions:

  • is there an option (or plan) to include --cap-add per service?
  • is there an option (or plan) to include --dns per service?
  • can I use service update to perform a service reload, without really changing stuff? for dev environment, I keep config files on volumes
  • what if a service has none restart policy, and it's sole task exits (gracefully or not)? can I reuse such service with service create or service update to make it running again?

Thank you

@thaJeztah
Member

@Garreat subscribe to #25303 to follow the discussion on additional options being added to services

can I use service update to perform a service reload, without really changing stuff? for dev environment, I keep config files on volumes

Looks like #24469

what if a service has none restart policy, and it's sole task exits (gracefully or not)? can I reuse such service with service create or service update to make it running again?

docker service update will always redeploy new containers if you make changes, so I think that should work

@Garreat
Garreat commented Aug 5, 2016

@thaJeztah that's good;

I'm looking forward for the functionality descripted in #24469 , --dns and --cap-add.

Also, I experience some mesh/dns misbehaviors descripted by @tectoro . Hopefully fixed in 1.12.1.
Nothing left but to wait.

I'm ok with closing this thread. Thanks again, the beer is on me!

@thaJeztah
Member

Cheers! 🍻

@fideloper
fideloper commented Aug 8, 2016 edited

I could use some clarification, as I'm running into this issue and it sounds like there's a lot of red herrings going on here (in particular improper use of the ingress network).

Here's my situation:

  • On AWS, with all ports open inside a default VPC (no overlapping IP ranges used between the VPC and docker's overlay network)
  • I'm not using the ingress network, but my own overlay network
  • I do have the same issue - inspecting my overlay network only shows 1 container per host I run docker network inspect <overlay-network-name on - each master and worker within the swarm.

Is this issue potentially fixed in 1.12.1 via #24789 ? I'm not sure that really described the issue - poor network performance is a bit different from a host just not appearing within an overlay network.

Additionally the official documentation appears to be behind 1.12 - We don't actually need an external key-value store any longer, correct?

If I'm reading the above right, it's basically saying to use a load balancer to balance between each host in the swarm, and circumvent Docker's internal load balancing - am I reading that right?

EDIT

I got this working on AWS by also opening up all connections and protocols in the AWS security group. See here for docs on ports that need to be open and able to communicate, although these might be slightly old for version 1.12.

@Garreat
Garreat commented Aug 8, 2016 edited

In my case, I nginx balance service app traffic and service www traffic in a Swarm (that is, container replicas of a particular service). For this purpose, I set up another lb service (nginx).
I could skip lb service and publish app and www straight away (1.12 stuff), BUT. As @mrjana pointed, I still want something like nginx for SSL offload - I use this. Otherwise, all www's tasks would have to perform in HTTPS mode.
Now, for lb's nginx.conf I define upstreams:

    upstream apps {
        server app:14444;
    }

    upstream workers {
        server www:80;
    }

SSL termination part:

        location / {
            proxy_pass http://workers;
        }

etc.

This syntax is enough, as I run www and app services in VIP (virtual IP) mode. It's enough to point at app, and Docker 1.12 will use IPVS mechanisms to load balance it further transparently. In the end, I run lb as published service + node constraints. Not perfect, but easy to integrate with anything.
An alternative is to use (old) DNSRR mode -- there is no VIP, you get the dynamic upstream (replica, task) list everytime you resolve. As non-commercial nginx resolves upstreams only at daemon up, I went for VIP. Also, you might want to google this The effect of RFC3484 on DNS Round Robin @medium to discover some built-in flaws of DNS load balancing.

Keep in mind that my environment is more classic static VMware / physical farm. No shiny AWS load balancer here ;). Btw for now it's clear that once you fiddle with service update, the mesh network misbehaves. Fingers crossed for 1.12.1!

This new swarm mode is different to the old one. You select an approach:

  • new: nodes, services, DAB, VIP/DNSRR, no consul required - raft built in
  • old: engines, containers, Compose, DNSRR, external KV

Another indirect implication is... if you want your application to pull any cluster state info, you place it on manager node and mount docker.sock. Possibly someone else could elaborate on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment