Swarm overlay network not routing via service VIP (18.09.4/ARM64) #644

johnmccabe · 2019-04-07T11:19:59Z

This is a bug report
This is a feature request
I searched existing issues before opening this one

Expected behavior

Deploying a stack on Swarm should result in services being accessible via the service VIP

Actual behavior

Service VIP does not route to container IPs, connecting to container IP works, setting endpoint_mode to dnsrr works.

The behaviour seems indentical to that observed in #525 but its not clear from that issue what the actual root cause was, what modules might I need to compile.

Steps to reproduce the behavior

Output of docker version:

Client:
 Version:           18.09.4
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        d14af54
 Built:             Wed Mar 27 18:41:13 2019
 OS/Arch:           linux/arm64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.4
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d14af54
  Built:            Wed Mar 27 18:02:08 2019
  OS/Arch:          linux/arm64
  Experimental:     false

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.4
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
 NodeID: ti27ibqqdiixsfl5zy64glxey
 Is Manager: true
 ClusterID: y0svczsqw10w14eizq3bigns4
 Managers: 1
 Nodes: 1
 Default Address Pool: 10.0.0.0/8
 SubnetSize: 24
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.10.10.154
 Manager Addresses:
  10.10.10.154:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.140-tegra
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.864GiB
Name: jetson
ID: ORDQ:3LM4:GF7B:LBRD:OEBC:DXF3:EYZZ:BSMV:YUZ5:NERW:RUQT:QKHX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.)
Single node swarm running on a Jetson Nano board, fresh docker install with no customisation.

Check config output

$ ./check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Reproducing the issue

Deploy simple stack.

$ docker stack deploy mystack -c ./docker-compose.yml

Where docker-compose.yml contains.

version: "3.7"
services:
    nginx:
        image: nginx
        networks:
            - mynet
        deploy:
            mode: replicated
            replicas: 1

networks:
    mynet:
        driver: overlay
        attachable: true

Check container and VIP IP addresses.

$ docker service inspect --format '{{json .Endpoint.VirtualIPs}}' mystack_nginx | jq .
[
{
    "NetworkID": "8t44va0sgf8y41zx4uvmq76r6",
    "Addr": "10.0.5.2/24"
}
]

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
5a37a41ccddd        nginx:latest        "nginx -g 'daemon of…"   4 minutes ago       Up 4 minutes        80/tcp              mystack_nginx.1.yrrvizn6r7yfsf1ud6hva86n8

$ docker container inspect 5a37a41ccddd  --format '{{json .NetworkSettings.Networks.mystack_mynet.IPAddress}}'
"10.0.5.3"

Type	IP
VIP	10.0.5.2
Container	10.0.5.3

Run container attached to same network (install curl).

$ docker run --network mystack_mynet -ti --rm alpine sh

Check direct container (10.0.5.3) connectivity - OK

/ # curl -I 10.0.5.3
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Sun, 07 Apr 2019 11:04:15 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 26 Mar 2019 14:04:38 GMT
Connection: keep-alive
ETag: "5c9a3176-264"
Accept-Ranges: bytes

Check VIP (10.0.5.2) connectivity - FAILS

/ # curl -I 10.0.5.2
curl: (7) Failed to connect to 10.0.5.2 port 80: Connection refused

The text was updated successfully, but these errors were encountered:

johnmccabe · 2019-04-07T11:35:09Z

Syslog output, not quite sure what I should be seeing here though and whether the device removal is expected?

deploying the stack.

https://gist.github.com/johnmccabe/61ccaf2136375269e1d425edd5850da9

attaching alpine container to network

https://gist.github.com/johnmccabe/68d649cc7f4c7601922b59ddbe7df066

justincormack · 2019-04-07T15:33:51Z

Is there anything in the daemon log (in debug mode)?

justincormack · 2019-04-07T15:48:42Z

There are lots of messages from NetworkManager trying to configure network settings on the interfaces. I tend to disable it on servers, limiting which network interfaces it looks at can alsohelp.

johnmccabe · 2019-04-11T23:21:53Z

Just got round to looking at this again tonight, recompiling the kernel did the trick..

49c49
< CONFIG_LOCALVERSION=""
---
> CONFIG_LOCALVERSION="-mccabe-v4"
733c733
< # CONFIG_INET_ESP is not set
---
> CONFIG_INET_ESP=y
826c826
< # CONFIG_NF_NAT_REDIRECT is not set
---
> CONFIG_NF_NAT_REDIRECT=m
857c857
< # CONFIG_NETFILTER_XT_TARGET_REDIRECT is not set
---
> CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
925,926c925,926
< # CONFIG_IP_VS_PROTO_TCP is not set
< # CONFIG_IP_VS_PROTO_UDP is not set
---
> CONFIG_IP_VS_PROTO_TCP=y
> CONFIG_IP_VS_PROTO_UDP=y
955a956
> # CONFIG_IP_VS_FTP is not set
956a958
> # CONFIG_IP_VS_PE_SIP is not set
983c985
< # CONFIG_IP_NF_TARGET_REDIRECT is not set
---
> CONFIG_IP_NF_TARGET_REDIRECT=m
1136c1138
< # CONFIG_NET_L3_MASTER_DEV is not set
---
> CONFIG_NET_L3_MASTER_DEV=y
1842a1845,1848
> CONFIG_DM_BUFIO=y
> # CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
> CONFIG_DM_BIO_PRISON=y
> CONFIG_DM_PERSISTENT_DATA=y
1845c1851
< # CONFIG_DM_THIN_PROVISIONING is not set
---
> CONFIG_DM_THIN_PROVISIONING=y
1877a1884
> CONFIG_IPVLAN=m
1889a1897
> # CONFIG_NET_VRF is not set
6215c6223
< CONFIG_CRYPTO_AUTHENC=m
---
> CONFIG_CRYPTO_AUTHENC=y
6231c6239
< CONFIG_CRYPTO_CBC=m
---
> CONFIG_CRYPTO_CBC=y
6257c6265
< CONFIG_CRYPTO_MD5=m
---
> CONFIG_CRYPTO_MD5=y
6280c6288
< CONFIG_CRYPTO_DES=m
---
> CONFIG_CRYPTO_DES=y
6364c6372
< CONFIG_LIBCRC32C=m
---
> CONFIG_LIBCRC32C=y

Enabling IPVLAN (and CONFIG_NET_L3_MASTER_DEV) on its own hadn't been working at the weekend, I enabled more of the missing modules as highlighted by check_config and it seems to have done the trick.

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

johnmccabe · 2019-04-11T23:22:37Z

Many thanks for the help and pointers @justincormack / @alexellis

johnmccabe · 2019-04-11T23:23:48Z

/ # nc nginx 80
GET /
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Points out another symbol that Docker might need. in this case Docker's mesh network in swarm mode does not route Virtual IPs if it's unset. From /var/logs/docker.log: time="2021-02-19T18:15:39+01:00" level=error msg="set up rule failed, [-t mangle -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257]: (iptables failed: iptables --wait -t mang le -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257: iptables v1.8.7 (legacy): unknown option \"--set-mark\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))" Bug: moby/libnetwork#2227 Bug: docker/for-linux#644 Bug: docker/for-linux#525 Signed-off-by: Piotr Karbowski <piotr.karbowski@protonmail.ch>

Points out another symbol that Docker might need. in this case Docker's mesh network in swarm mode does not route Virtual IPs if it's unset. From /var/logs/docker.log: time="2021-02-19T18:15:39+01:00" level=error msg="set up rule failed, [-t mangle -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257]: (iptables failed: iptables --wait -t mang le -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257: iptables v1.8.7 (legacy): unknown option \"--set-mark\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))" Bug: moby/libnetwork#2227 Bug: docker/for-linux#644 Bug: docker/for-linux#525 Signed-off-by: Piotr Karbowski <piotr.karbowski@protonmail.ch> Upstream-commit: e8ceb976469e15547ed368ba5c110102ccc5fbfa Component: engine

Points out another symbol that Docker might need. in this case Docker's mesh network in swarm mode does not route Virtual IPs if it's unset. From /var/logs/docker.log: time="2021-02-19T18:15:39+01:00" level=error msg="set up rule failed, [-t mangle -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257]: (iptables failed: iptables --wait -t mang le -A INPUT -d 10.0.1.2/32 -j MARK --set-mark 257: iptables v1.8.7 (legacy): unknown option \"--set-mark\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))" Bug: moby/libnetwork#2227 Bug: docker/for-linux#644 Bug: docker/for-linux#525 Signed-off-by: Piotr Karbowski <piotr.karbowski@protonmail.ch> (cherry picked from commit e8ceb97) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

johnmccabe closed this as completed Apr 13, 2019

slashbeast mentioned this issue Feb 19, 2021

check-config.sh: add NETFILTER_XT_MARK moby/moby#42052

Merged

thaJeztah mentioned this issue Feb 23, 2021

[20.10 backport] check-config.sh: add NETFILTER_XT_MARK moby/moby#42066

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Swarm overlay network not routing via service VIP (18.09.4/ARM64) #644

Swarm overlay network not routing via service VIP (18.09.4/ARM64) #644

johnmccabe commented Apr 7, 2019

johnmccabe commented Apr 7, 2019

justincormack commented Apr 7, 2019

justincormack commented Apr 7, 2019

johnmccabe commented Apr 11, 2019

johnmccabe commented Apr 11, 2019

johnmccabe commented Apr 11, 2019

Swarm overlay network not routing via service VIP (18.09.4/ARM64) #644

Swarm overlay network not routing via service VIP (18.09.4/ARM64) #644

Comments

johnmccabe commented Apr 7, 2019

Expected behavior

Actual behavior

Steps to reproduce the behavior

johnmccabe commented Apr 7, 2019

justincormack commented Apr 7, 2019

justincormack commented Apr 7, 2019

johnmccabe commented Apr 11, 2019

johnmccabe commented Apr 11, 2019

johnmccabe commented Apr 11, 2019