Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to 20.10 breaks swarm network #41775

Open
oneumyvakin opened this issue Dec 10, 2020 · 65 comments
Open

Upgrade to 20.10 breaks swarm network #41775

oneumyvakin opened this issue Dec 10, 2020 · 65 comments

Comments

@oneumyvakin
Copy link

oneumyvakin commented Dec 10, 2020

Description

Steps to reproduce the issue:

  1. Install Docker 19.03 on Ubuntu 20 or CentOS 8
  2. Init Swarm
  3. Start some services by docker stack deploy
  4. Upgrade docker from 19.03 to 20.10

Describe the results you received:
Containers of services can't start
there is error in logs

Dec 10 06:21:03 dockerd[3160859]: time="2020-12-10T06:21:03.150920367Z" level=error msg="fatal task error" error="starting container failed: container 9f93a21ac2e3be11a65c91f3cfde555a415eea47c636bef432d5d2e4b08afff4: endpoint create on GW Network failed: failed to create endpoint gateway_f8cabe848464 on network docker_gwbridge: network 28d599d44202f2acdc85e42437332ddb41a81bd7f0622bc0724761ec9b49082a does not exist" module=node/agent/taskmanager node.id=u7qdqny1doho69k3nariuo1ru service.id=vhtg6aoyt360k7mluiwmshqf0 task.id=aqgsfcw88m5kujnrly74o4wh4

Describe the results you expected:
containers are running

Additional information you deem important (e.g. issue happens only occasionally):
we have two installations with this issue which happened after upgrade to 20.10
recreating services didn't help
re-initing swarm didn't help

# docker network list
NETWORK ID     NAME              DRIVER    SCOPE
e45b9b63c4ae   bridge            bridge    local
28d599d44202   docker_gwbridge   bridge    local
2aa80dc0cc04   host              host      local
w9rpuika2x0d   ingress           overlay   swarm
62f0fb2fdf28   none              null      local

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:40 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: u7qdqny1doho69k3nariuo1ru
  Is Manager: true
  ClusterID: rdq2vi44m2lkz34tdow1dvip4
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 127.0.0.1
  Manager Addresses:
   127.0.0.1:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-56-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.749GiB
 Name: cloud.filesanctuary.net
 ID: JBWW:XVUE:3XW4:OQYT:HJHK:OSRV:PFHK:PFZP:S3DV:HPZ7:NYWD:OWQO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
@thaJeztah
Copy link
Member

Thanks for reporting; I just tried updating a test-cluster I had running on docker 19.03, but it didn't reproduce on that cluster; if you can find more details (perhaps daemon logs contain useful information?) that would be appreciated.

/cc @arkodg @dperny

@thaJeztah thaJeztah added this to the 20.10.1 milestone Dec 10, 2020
@oneumyvakin
Copy link
Author

oneumyvakin commented Dec 10, 2020

OK, I've resolved the issue, but not sure what was the root cause.

I've tried to downgrade docker and containerd to 19.03 and 1.3.7, but it didn't helps.

I've spotted that docker starts using firewalld to add interfaces docker0 and docker_gwbridge to "docker" zone.

On our installations we've added docker_gwbridge in "trusted" zone.

To resolve the issue I've to:

  • remove docker_gwbridge from "trusted" zone in firewalld
  • delete "docker" zone in firewalld
  • deinstall Docker and containerd
  • delete folder /var/lib/docker/network/files/
  • install latest version of docker and containerd

@Nowheresly
Copy link

Nowheresly commented Dec 14, 2020

We experienced the same issue on our swarm cluster (OS: ubuntu 16.04, no firewalld):

From the daemon log of a worker node, I can see:

dockerd[2228]: time="2020-12-14T16:38:2902148446+01:00" level="warning" msg="[resolver] connect failed: dial udp:XX.XX.XX.XX:53: connect: network is unreachable"

We don't have this error on manager nodes.

@Nowheresly
Copy link

Ok, after re-reading the issue,we don't experience the same issue. Our containers start but they randomly fail see each other (unexpected EOF, connection closed) even if they are in the same network (driver: overlay).

@thaJeztah thaJeztah modified the milestones: 20.10.1, 20.10.2 Dec 15, 2020
@oneumyvakin
Copy link
Author

Looks like this issue actual for CentOS 8 as well.

@sgohl
Copy link

sgohl commented Jan 29, 2021

any news here? I have the same problem on a fresh CentOS8.3 install (also Stream, from netinstall)

dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf -y install docker-ce

> docker -v
Docker version 20.10.2, build 2291f61

> cat /etc/*release*
CentOS Linux release 8.3.2011

> rpm -qa | grep docker-ce
docker-ce-rootless-extras-20.10.2-3.el8.x86_64
docker-ce-cli-20.10.2-3.el8.x86_64
docker-ce-20.10.2-3.el8.x86_64

> rpm -qa | grep containerd
containerd.io-1.4.3-3.1.el8.x86_64

> uname -r
4.18.0-269.el8.x86_64

 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
  Data Path Port: 4789
  Default Address Pool: 10.0.0.0/8  
  Network: bridge host ipvlan macvlan null overlay
 Cgroup Driver: cgroupfs
 Cgroup Version: 1

after swarm init (3 managers, no workers), creating a test service

docker service create --name nginx --publish 88:80 --constraint 'node.role == manager' nginx:alpine

curl always fails on every node except the node where the replica is actually running

disabled selinux (permissive) and firewalld to eliminate as reason
have no custom daemon.json applied

@ghost
Copy link

ghost commented Feb 1, 2021

I'm experiencing this issue as well, pretty much similar to all of the errors/specifications above, running Ubuntu 20.04 LTS and to say the least it cripples my workflow 🤒.

Are there any news on that/ETA?

@thaJeztah thaJeztah modified the milestones: 20.10.3, 20.10.4 Feb 2, 2021
@sgohl
Copy link

sgohl commented Feb 3, 2021

interestingly this does not happen with the vagrant image boxomatic/centos-8-stream although all versions I can overview are the same after full system upgrade.
Also, a cloud vm installation on hetzner did also not have this issue despite the same versions.
I only experience that issue on two different VMware ESX vSphere (6.7) environments. I began to think it's a networking issue on my side, but other VMs on the same hypervisors do not have that problem.
I'm clueless, debugging swarm ingress networking isn't very straight forward for me, unfortunately

Did anything change regarding the default networks used for bip, gw or ingress?

I made multiple tests installing old versions of containerd.io, docker-ce 19.03 and matching versions from ~august 2020, but still the same

@spacepirate0001
Copy link

spacepirate0001 commented Feb 4, 2021

Same issue Centos7, after upgrade 19.03.14 to 20.10.3. Services lose connectivity between each other in swarm mode. I tried removing /var/lib/docker/network/files/* and restart swarm but no luck.

@jjvalent
Copy link

Is there an update on this issue? I am facing the same on centos 7 and centos 8.

@sgohl
Copy link

sgohl commented Feb 10, 2021

Is there an update on this issue? I am facing the same on centos 7 and centos 8.

could you (all affected) perhaps provide some debugging to investigate whether we all actually facing the same problem?

  • custom /etc/docker/daemon.json config, if any
  • Networks of bridge ip, docker_gw, ingress
  • create basic service attached to swarm mesh (see Upgrade to 20.10 breaks swarm network #41775 (comment))
  • docker service inspect nginx | jq network output from this service
  • check multiple curl swarm-ip to prove it always only responds where nginx is actually running
  • create a stack of 2 services, then bash into one and curl/ping the other and the service-vip (and provide container ips and service vip)

Any cloud provider I try (aws), there it works, also vagrant, but where I actually need it (onpremise vsphere, VM installed via PXE/8-stream) it doesnt work.

unfortunately I have no bare metal available to install 3 systems from a naked ISO (newest centos 8 stream at best). that would be a very interesting test, if anyone could do this?

@Nowheresly
Copy link

I use swarm and I had network connectivity issues right after migrating to docker 20.10.x.
After struggling a bit, I was able to find out the problem and to fix it.

I use overlay networks for my swarm services and it's very common that my services are defined in several networks. So it basically means that my service have several ips (one for each of its network).

In the example below, my nginx has one hostname (server1) but at least 2 ips (ip1 in network net1 and ip2 in network net2).

services:
  nginx:
    image: nginx:latest
    hostname: server1
    network:
      - net1
      - net2

Now, here comes the interesting part: docker 19.03.x and docker 20.10.x behave differently when it comes to resolve the ip of the host server1.

docker 19.03.x ALWAYS returns the same ip (which can be either ip1 or ip2 in my above example)

whereas docker 20.10.x returns ALTERNATIVELY ip1 and ip2 (round-robin).

Now my problem was that I was using hostnames in my services and then I was using GO primitives such as
ResolveTCPAddr to get the ip and connect to other services and I used lib such as pollon (https://github.com/sorintlab/pollon/blob/248c68238c160c056cd51d0e782276cef5c64ce4/pollon.go#L130) to track ip changes and reinit connections each time an ip change was detected...

So, since docker 20.10 is now returning a different ip after each DNS request when services have several ips, I was endlessly losing connection...

After realizing this, I had to modify the code of my services to take into account this new behavior.

I don't know if what I'm describing here could be related to this issue.
I'm just giving my feedback of the issues I had during my docker migration in case it may help someone...

@spacepirate0001
Copy link

spacepirate0001 commented Feb 11, 2021

I use swarm and I had network connectivity issues right after migrating to docker 20.10.x.
After struggling a bit, I was able to find out the problem and to fix it.

I use overlay networks for my swarm services and it's very common that my services are defined in several networks. So it basically means that my service have several ips (one for each of its network).

In the example below, my nginx has one hostname (server1) but at least 2 ips (ip1 in network net1 and ip2 in network net2).

services:
  nginx:
    image: nginx:latest
    hostname: server1
    network:
      - net1
      - net2

Now, here comes the interesting part: docker 19.03.x and docker 20.10.x behave differently when it comes to resolve the ip of the host server1.

docker 19.03.x ALWAYS returns the same ip (which can be either ip1 or ip2 in my above example)

whereas docker 20.10.x returns ALTERNATIVELY ip1 and ip2 (round-robin).

Now my problem was that I was using hostnames in my services and then I was using GO primitives such as
ResolveTCPAddr to get the ip and connect to other services and I used lib such as pollon (https://github.com/sorintlab/pollon/blob/248c68238c160c056cd51d0e782276cef5c64ce4/pollon.go#L130) to track ip changes and reinit connections each time an ip change was detected...

So, since docker 20.10 is now returning a different ip after each DNS request when services have several ips, I was endlessly losing connection...

After realizing this, I had to modify the code of my services to take into account this new behavior.

I don't know if what I'm describing here could be related to this issue.
I'm just giving my feedback of the issues I had during my docker migration in case it may help someone...

I think its plausible! In my case its a socket.io server service these need a locked connection tunnel and started losing connection after migration to 20.10. Also, I use service names as alias like in your case. The ALTERNATIVELY behavior described above would certainly break it.

@jjvalent
Copy link

jjvalent commented Feb 24, 2021

Is there an update on this issue? I am facing the same on centos 7 and centos 8.

could you (all affected) perhaps provide some debugging to investigate whether we all actually facing the same problem?

* custom `/etc/docker/daemon.json` config, if any

* Networks of bridge ip, docker_gw, ingress

* create basic service attached to swarm mesh (see [#41775 (comment)](https://github.com/moby/moby/issues/41775#issuecomment-769873804))

* `docker service inspect nginx | jq` network output from this service

* check multiple `curl swarm-ip` to prove it always only responds where nginx is actually running

* create a stack of 2 services, then bash into one and curl/ping the other and the service-vip (and provide container ips and service vip)

Any cloud provider I try (aws), there it works, also vagrant, but where I actually need it (onpremise vsphere, VM installed via PXE/8-stream) it doesnt work.

unfortunately I have no bare metal available to install 3 systems from a naked ISO (newest centos 8 stream at best). that would be a very interesting test, if anyone could do this?

Output of docker networks

# docker inspect bridge [ { "Name": "bridge", "Id": "ffd79f446e2db01624773bcca03273943e7840c00f7e1a722818a15fc4df1e9e", "Created": "2021-02-24T11:10:33.710699325-05:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ]

# docker inspect docker_gwbridge [ { "Name": "docker_gwbridge", "Id": "782eac048eced5098b6385cfd09907b0044139f778267046df2bfb1230cb0ffc", "Created": "2021-02-23T12:28:01.780059612-05:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.18.0.0/16", "Gateway": "172.18.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "7fab1e416ceff98ff5cfdc0755ebdad6addd7142c5213069a53d261b8d4c705f": { "Name": "gateway_6bbd9caf63c0", "EndpointID": "2669848c2ef5b01c61b0dae07961aabf5c358954ef5890dd49aeacf59c8864a6", "MacAddress": "02:42:ac:12:00:03", "IPv4Address": "172.18.0.3/16", "IPv6Address": "" }, "d4eeee7b07e78bffb712811f6742d24d1f63a0ef582dfe7ff74bab476425f2b6": { "Name": "gateway_d97df4f123eb", "EndpointID": "3cd5aad794d7b4c43afc2fd969833b6ba64792f25aa64dfbcd4a33f698e77bcc", "MacAddress": "02:42:ac:12:00:04", "IPv4Address": "172.18.0.4/16", "IPv6Address": "" }, "ingress-sbox": { "Name": "gateway_ingress-sbox", "EndpointID": "e34ef0a04d734acfe74732899edffb33e5a29ad6edf3abe979d388140ec71ac3", "MacAddress": "02:42:ac:12:00:02", "IPv4Address": "172.18.0.2/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.enable_icc": "false", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.name": "docker_gwbridge" }, "Labels": {} } ]

# docker network inspect ingress [ { "Name": "ingress", "Id": "mty0tbdrmvuqhvr444bvfgydd", "Created": "2021-02-24T11:10:35.14459346-05:00", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "10.0.0.0/24", "Gateway": "10.0.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": true, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "7fab1e416ceff98ff5cfdc0755ebdad6addd7142c5213069a53d261b8d4c705f": { "Name": "my-web.2.yh8cxiw301exaw2n3x1w8rs3l", "EndpointID": "64d2edb8126b55a328c01d08c54e796ed6e5ec6077a7735e41d5c5d967daf8a2", "MacAddress": "02:42:0a:00:00:05", "IPv4Address": "10.0.0.5/24", "IPv6Address": "" }, "d4eeee7b07e78bffb712811f6742d24d1f63a0ef582dfe7ff74bab476425f2b6": { "Name": "my-web.1.1aj142fcfz7ltg0h23pc8om42", "EndpointID": "f30f109f639fe5248ffc99dbc04bb5613ac3199e55879c7eea861495a5820b5d", "MacAddress": "02:42:0a:00:00:0a", "IPv4Address": "10.0.0.10/24", "IPv6Address": "" }, "ingress-sbox": { "Name": "ingress-endpoint", "EndpointID": "d4b5be804a3ac18477b1c14d7acf62133fbac997a96cec66de6939ed6a5a2db2", "MacAddress": "02:42:0a:00:00:02", "IPv4Address": "10.0.0.2/24", "IPv6Address": "" } }, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4096" }, "Labels": {}, "Peers": [ { "Name": "fb673deb11cb", "IP": "192.168.37.201" }, { "Name": "523e4d83611d", "IP": "192.168.37.202" }, { "Name": "40154e99b45c", "IP": "192.168.37.203" } ] } ]

docker service inspect my-web | jq [ { "ID": "8j2n9mzt9cb6nw9b0rjgx9shj", "Version": { "Index": 200 }, "CreatedAt": "2021-02-23T20:52:45.817797547Z", "UpdatedAt": "2021-02-24T16:10:34.144883727Z", "Spec": { "Name": "my-web", "Labels": {}, "TaskTemplate": { "ContainerSpec": { "Image": "nginx:latest@sha256:f3693fe50d5b1df1ecd315d54813a77afd56b0245a404055a946574deb6b34fc", "Init": false, "StopGracePeriod": 10000000000, "DNSConfig": {}, "Isolation": "default" }, "Resources": { "Limits": {}, "Reservations": {} }, "RestartPolicy": { "Condition": "any", "Delay": 5000000000, "MaxAttempts": 0 }, "Placement": { "Platforms": [ { "Architecture": "amd64", "OS": "linux" }, { "OS": "linux" }, { "OS": "linux" }, { "Architecture": "arm64", "OS": "linux" }, { "Architecture": "386", "OS": "linux" }, { "Architecture": "mips64le", "OS": "linux" }, { "Architecture": "ppc64le", "OS": "linux" }, { "Architecture": "s390x", "OS": "linux" } ] }, "Networks": [ { "Target": "1iem30yxy7aztaqzxtxgqxq95" } ], "ForceUpdate": 0, "Runtime": "container" }, "Mode": { "Replicated": { "Replicas": 4 } }, "UpdateConfig": { "Parallelism": 1, "FailureAction": "pause", "Monitor": 5000000000, "MaxFailureRatio": 0, "Order": "stop-first" }, "RollbackConfig": { "Parallelism": 1, "FailureAction": "pause", "Monitor": 5000000000, "MaxFailureRatio": 0, "Order": "stop-first" }, "EndpointSpec": { "Mode": "vip", "Ports": [ { "Protocol": "tcp", "TargetPort": 80, "PublishedPort": 8080, "PublishMode": "ingress" } ] } }, "PreviousSpec": { "Name": "my-web", "Labels": {}, "TaskTemplate": { "ContainerSpec": { "Image": "nginx:latest@sha256:f3693fe50d5b1df1ecd315d54813a77afd56b0245a404055a946574deb6b34fc", "Init": false, "DNSConfig": {}, "Isolation": "default" }, "Resources": { "Limits": {}, "Reservations": {} }, "Placement": { "Platforms": [ { "Architecture": "amd64", "OS": "linux" }, { "OS": "linux" }, { "OS": "linux" }, { "Architecture": "arm64", "OS": "linux" }, { "Architecture": "386", "OS": "linux" }, { "Architecture": "mips64le", "OS": "linux" }, { "Architecture": "ppc64le", "OS": "linux" }, { "Architecture": "s390x", "OS": "linux" } ] }, "Networks": [ { "Target": "1iem30yxy7aztaqzxtxgqxq95" } ], "ForceUpdate": 0, "Runtime": "container" }, "Mode": { "Replicated": { "Replicas": 2 } }, "EndpointSpec": { "Mode": "vip", "Ports": [ { "Protocol": "tcp", "TargetPort": 80, "PublishedPort": 8080, "PublishMode": "ingress" } ] } }, "Endpoint": { "Spec": { "Mode": "vip", "Ports": [ { "Protocol": "tcp", "TargetPort": 80, "PublishedPort": 8080, "PublishMode": "ingress" } ] }, "Ports": [ { "Protocol": "tcp", "TargetPort": 80, "PublishedPort": 8080, "PublishMode": "ingress" } ], "VirtualIPs": [ { "NetworkID": "mty0tbdrmvuqhvr444bvfgydd", "Addr": "10.0.0.4/24" }, { "NetworkID": "1iem30yxy7aztaqzxtxgqxq95", "Addr": "10.202.0.2/24" } ] } } ]

`# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
Containers: 8
Running: 2
Paused: 0
Stopped: 6
Images: 1
Server Version: 20.10.3
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: ibhoruoc3evvmsdi16u9cynef
Is Manager: true
ClusterID: wkijyum8cv944f34l02m2j0om
Managers: 1
Nodes: 3
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.37.201
Manager Addresses:
192.168.37.201:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.18.0-240.10.1.el8_3.x86_64
Operating System: CentOS Linux 8
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.21GiB
Name: ****
ID: NYJ3:D3X4:LGZE:SPN3:XLAK:TWUY:OYRB:U27K:GEAR:FAYH:2O4P:756L
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support
`

`root@d8766a2befac:/# ping my-web.1.1aj142fcfz7ltg0h23pc8om42 -c 3
PING my-web.1.1aj142fcfz7ltg0h23pc8om42 (10.202.0.10) 56(84) bytes of data.
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=1 ttl=64 time=0.606 ms
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=2 ttl=64 time=0.431 ms
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=3 ttl=64 time=0.365 ms

--- my-web.1.1aj142fcfz7ltg0h23pc8om42 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 50ms
rtt min/avg/max/mdev = 0.365/0.467/0.606/0.103 ms
root@d8766a2befac:/# curl http://my-web.1.1aj142fcfz7ltg0h23pc8om42:8080/
^C
root@d8766a2befac:/# curl http://my-web.1.1aj142fcfz7ltg0h23pc8om42:8080/ --max-time 15
curl: (28) Connection timed out after 15001 milliseconds
root@d8766a2befac:/# ping my-web.1.1aj142fcfz7ltg0h23pc8om42 -c 3
PING my-web.1.1aj142fcfz7ltg0h23pc8om42 (10.202.0.10) 56(84) bytes of data.
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=1 ttl=64 time=0.469 ms
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=2 ttl=64 time=0.418 ms
64 bytes from my-web.1.1aj142fcfz7ltg0h23pc8om42.nginx-services (10.202.0.10): icmp_seq=3 ttl=64 time=0.444 ms

--- my-web.1.1aj142fcfz7ltg0h23pc8om42 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 81ms
rtt min/avg/max/mdev = 0.418/0.443/0.469/0.032 ms
root@d8766a2befac:/# curl http://my-web.1.1aj142fcfz7ltg0h23pc8om42:8080/ --max-time 15
curl: (28) Connection timed out after 15001 milliseconds
root@d8766a2befac:/# dig my-web.1.1aj142fcfz7ltg0h23pc8om42

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Debian <<>> my-web.1.1aj142fcfz7ltg0h23pc8om42
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17813
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;my-web.1.1aj142fcfz7ltg0h23pc8om42. IN A

;; ANSWER SECTION:
my-web.1.1aj142fcfz7ltg0h23pc8om42. 600 IN A 10.202.0.10

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Wed Feb 24 17:25:37 UTC 2021
;; MSG SIZE rcvd: 102

root@d8766a2befac:/# traceroute my-web.1.1aj142fcfz7ltg0h23pc8om42
traceroute to my-web.1.1aj142fcfz7ltg0h23pc8om42 (10.202.0.10), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
root@d8766a2befac:/# `

@jjvalent
Copy link

I have added more info above, the my-web docker swarm service is taken from the swarm nginx example.

@txtdevelop
Copy link

txtdevelop commented Feb 25, 2021

We also have connectivity problems in our docker swarm (3 redhat 8.3 vm nodes)
The services running in the containers are not accessible using the swarm mode routing mesh but only using the explicit host ip

After some investigation, we found that the problem is related to the 4789 udp packets that docker uses to manage the requests in the swarm: these packets are dropped by the source node and they never reach the destinatation node

To resolve this issue we had to disable the following offload feature:

ethtool -K [network] tx-checksum-ip-generic off

update:
similar problem flannel-io/flannel#1279

@thaJeztah thaJeztah removed this from the 20.10.4 milestone Feb 25, 2021
@alejandroSanchezCaceres

I can confirm this is exactly the solution for at least my case (centos 8.3 stream, docker 20.10.5)

ethtool -K ens192 tx-checksum-ip-generic off

after executing on all swarm machines, routing mesh now works! This seems to be reboot-safe
cheers @txtdevelop !
Edit/ps: it may be reboot-safe, but after recent dnf update the setting was lost again.
For anyone needing it: (ETHTOOL_OPTS= seems not recognized in centos8-stream when using NM)

cat > /etc/NetworkManager/dispatcher.d/pre-up.d/10-tx-checksum-ip-generic <<'EOF'
ethtool -K ens192 tx-checksum-ip-generic off
EOF
chmod +x /etc/NetworkManager/dispatcher.d/pre-up.d/10-tx-checksum-ip-generic

makes it persistent
check:

ethtool --show-offload ens192 | grep tx-checksum-ip-generic

Works for us on the Docker Swarm worker node with CentOS 8.3 and Docker 20.10.5 Thank you @sgohl

For me this works fine with RHEL 8.4

@shakibamoshiri
Copy link

shakibamoshiri commented Jan 29, 2022

problem with Docker version 20.10.12, build e91ed57 on:

Debian GNU/Linux 11 (bullseye)
Linux master 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux

and seems to be solved with

ethtool -K   ens192   tx-checksum-ip-generic off

no problem with Docker version 20.10.12, build e91ed57 on

Ubuntu 20.04.3 LTS (Focal Fossa)
Linux master 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

@kleptog
Copy link

kleptog commented Mar 18, 2022

We've run into this issue as well. The strange thing is we had two essentially identical environments, one has the issue, the other works fine.

These are the package versions we're using, but it's probably not that since one environment has this and it works.

# dpkg -l container* docker* linux-image* |grep ^ii
ii  containerd.io                        1.4.13-1                       amd64        An open and reliable container runtime
ii  docker-ce                            5:20.10.11~3-0~debian-bullseye amd64        Docker: the open-source application container engine
ii  docker-ce-cli                        5:20.10.11~3-0~debian-bullseye amd64        Docker CLI: the open-source application container engine
ii  linux-image-5.10.0-11-amd64          5.10.92-2                      amd64        Linux 5.10 for 64-bit PCs (signed)

The symptom is that the overlay network doesn't work. The way to test this is with tcpdump:

# tcpdump -i ens224 -n port 4789 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens224, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:46:04.028007 IP 10.xx.xx.1.34013 > 10.xx.xx.2.4789: VXLAN, flags [I] (0x08), vni 40xx
IP 10.xx.xx.6.35616 > 10.xx.xx.7.5555: Flags [S], seq 1429907708, win 64860, options [mss 1410,sackOK,TS val 974838813 ecr 0,nop,wscale 7], length 0

When it's broken you only see packets going out, but no packets coming in. You need to have some containers running to trigger overlay traffic.

In our case we needed to do

ethtool -K ens224 tx-checksum-ip-generic off

on all swarm hosts. We added it to /etc/network/interfaces pre-up to fix this.

The only difference we've found is that one environment is VM version 13 (which works) and the other is VM version 14 (which doesn't). We've found reference to a VMWare PR 2766401 which refers to a bug causing the vmxnet driver to drop packets. This is apparently fixed from VM version 15.

So our hypothesis is that if you running VM version 14 with the Debian 5.10.92-2 kernel it breaks, but running an older kernel version (in our case 4.19.98-1+deb10u1) or an older VM version, it works fine.

For reference, these versions worked everywhere for us (prior to upgrading)

ii  containerd.io                       1.4.4-1                      amd64        An open and reliable container runtime
ii  docker-ce                           5:19.03.3~3-0~debian-stretch amd64        Docker: the open-source application container engine
ii  docker-ce-cli                       5:19.03.3~3-0~debian-stretch amd64        Docker CLI: the open-source application container engine
ii  linux-image-4.19.0-8-amd64          4.19.98-1+deb10u1            amd64        Linux 4.19 for 64-bit PCs (signed)

For reference, we were running VMware ESXi, 7.0.3, 19193900 on both environments.

@dejeroWilliamScott
Copy link

dejeroWilliamScott commented Mar 25, 2022

We recently encountered a similar issue on Azure, i.e. containers could ping containers on other nodes, but other traffic would hang indefinitely (e.g. curl, mysql, etc.). Our issue was resolved by a different solution than those presented in this thread, so I'm posting it here for completeness/awareness.

We RCA'd our issue to an Ubuntu Kernel update, specifically;

# functional
Linux test-vm-4 5.4.0-1072-azure #75~18.04.1-Ubuntu SMP Wed Mar 2 14:41:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 
# breaks cross-node traffic
Linux test-vm-5 5.4.0-1073-azure #76~18.04.1-Ubuntu SMP Thu Mar 10 11:17:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 

Downgrading the Kernel to 5.4.0-1072 (shown removing the 1073 version) restored cross-node container connectivity,

 sudo apt remove \
  linux-azure-5.4-cloud-tools-5.4.0-1073 \
  linux-azure-5.4-headers-5.4.0-1073 \
  linux-azure-5.4-tools-5.4.0-1073 \
  linux-modules-5.4.0-1073-azure \
  linux-modules-extra-5.4.0-1073-azure
sudo reboot

We initially thought the connectivity loss was related to a Docker Swarm upgrade (specifically to 20.10). However, we later determined that it wasn't the Docker upgrade at all -- it was the reboot that we performed while doing it (which loaded the new Kernel on our test environments).

Edit -- the new Kernel (1073) went live earlier this week (2022-03-22*)

@lightningspirit
Copy link

lightningspirit commented Mar 26, 2022

After upgrading to Linux Kernel to 5.4.0-105-generic on Ubuntu 20.04, the same thing happened to us in one node. @dejeroWilliamScott experience is similar to ours. Containers have connectivity with other containers on the same overlay network, however connection to the swarm on published ports (such as Traefik public ports), do not connect. There's although one thing that should be considered. In our case the ingress overlay network was created with encrypted option enabled. Take in consideration that created timestamp in this particular node after reboot changed.

$ docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "k9sgxk26p1l2z25751gue64ew",
        "Created": "2022-03-25T01:28:38.257556887Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.1.0.0/16",
                    "Gateway": "10.1.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4111",
            "com.docker.network.mtu": "1500",
            "encrypted": "true"
        }
    }
]

EDIT: will try to recreate the ingress network with default options.

@regbo
Copy link

regbo commented Mar 31, 2022

I had automatic updates enabled on my manager nodes, and out of nowhere they started consistently failing overnight last week. I believe it's because they automtaically updated to a 5+ kernel. I have tried disabling kernel updates, and will post my findings.

@lightningspirit
Copy link

Following my comment above we recreated the ingress overlay network without the encrypted option (only the default options). We also recreated all networks that were using the encrypted option.
After all configured we started the upgrade of kernel on each node one by one. Everything worked as expected. Without much proves, I'm inclined to say that there's an incompatibility with the built-in encryption on overlay networks and their recreation after docker or kernel upgrades.

Btw, just for further reference, since this is a multi-cloud instance without VPC, we had to encrypt the --data-path-addr by using the good vpncloud interface instead of the built-in encryption.

@lightningspirit
Copy link

I had automatic updates enabled on my manager nodes, and out of nowhere they started consistently failing overnight last week. I believe it's because they automtaically updated to a 5+ kernel. I have tried disabling kernel updates, and will post my findings.

@regbo are your overlay networks configured with the encryption option?

@dejeroWilliamScott
Copy link

dejeroWilliamScott commented Mar 31, 2022

I had automatic updates enabled on my manager nodes, and out of nowhere they started consistently failing overnight last week. I believe it's because they automtaically updated to a 5+ kernel. I have tried disabling kernel updates, and will post my findings.

@regbo are your overlay networks configured with the encryption option?

I've failed to reproduce using unencrypted networks (and successfully reproduced using encrypted networks). I think it is safe to conclude that there is a correlation between encrypted networks and the Kernel update. I've attached a simplified stack.yaml file for reference.

version: '3.7'

networks:
  encrypted:
    attachable: true
    driver: overlay
    driver_opts:
      encrypted: ""
  unencrypted:
    attachable: true
    driver: overlay

services:
  encrypted:
    image: "nginxdemos/hello"
    deploy:
      mode: global
    networks:
      - encrypted
    stdin_open: true
    tty: true
  unencrypted:
    image: "nginxdemos/hello"
    deploy:
      mode: global
    networks:
      - unencrypted
    stdin_open: true
    tty: true

I've created a new issue (#43443) to prevent confounding the two potentially different issues here. If it is later determined to be the same issue, we can rejoin them then.

@UchihaYuki
Copy link

@jmcombs
Copy link

jmcombs commented Apr 24, 2022

I am having this issue, as well running Photon OS. My issue seems to be related to something with regards to iptables. Overlay networking and everything works fine but inbound routing mesh. I disable iptables issue goes away. I'm not real good with debugging iptables so I am still limping through this. vmware/photon#1321

@bluepuma77
Copy link

bluepuma77 commented Jul 1, 2022

Just added a new server to my Docker Swarm and suddenly had issues with the unencrypted overlay network, which runs over a VLAN with 1400 MTU without any special settings. After some experimentation it turns out that it seems to be a new kernel (or Debian) problem, but not necessarily Docker Swarm itself.

Those "distributions" work fine with latest Docker Swarm:

Linux hostname 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
Linux hostname 4.19.0-17-amd64 #1 SMP Debian 4.19.194-1 (2021-06-10) x86_64 GNU/Linux
Linux hostname 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
Linux hostname 4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64 GNU/Linux
Linux hostname 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Linux hostname 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

With those the network had an issue with responses larger ~1400 bytes:

Linux hostname 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
Linux hostname 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 GNU/Linux

Let me add two thoughts:

  1. I don't believe it's a firewall problem. Firewalls usually forward or block, but they don't selectively drop connections when more data is transferred. I had issues with short requests working and long ones failing before, and of course it was an issue related to MTU size.
  2. Strangely enough the Docker Swarm overlay network does work on older kernels, despite transferring data over a VLAN with smaller 1400 MTU. Hypothesis: Docker Swarm checks the MTU of the connection and adjusts it internally if necessary. But with the new kernel something changed and therefore the recognition of MTU fails.

@RicardoViteriR
Copy link

RicardoViteriR commented Aug 12, 2022

Hello, I believed I faced this issue when I upgraded to 20.10.17 on Ubuntu 22 Jammy. The network was blocking all requests to containers. I was able to fix the issue by downgrading to 20.10.17 using apt-get install docker-ce=5:20.10.16~3-0~ubuntu-jammy docker-ce-cli=5:20.10.16~3-0~ubuntu-jammy containerd.io docker-compose-plugin

@troman29
Copy link

troman29 commented Oct 12, 2022

Hello, I believed I faced this issue when I upgraded to 20.10.17 on Ubuntu 22 Jammy. The network was blocking all requests to containers. I was able to fix the issue by downgrading to 20.10.17 using apt-get install docker-ce=5:20.10.16~3-0~ubuntu-jammy docker-ce-cli=5:20.10.16~3-0~ubuntu-jammy containerd.io docker-compose-plugin

@RicardoViteriR Thank you, this is the only solution that helped me!

@troman29
Copy link

@RicardoViteriR Thank you, this is the only solution that helped me!

Unfortunately, it still doesn't work. :(

@RicardoViteriR Maybe you know other reasons? The problem is the same, containers between nodes are unavailable (timeout), DNS works correctly.

# MANAGER
root@infra-1:~# docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:15:28 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@infra-1:~# uname -as
Linux infra-1.obbana.ru 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# WORKER
root@infra-2:~# docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:15:28 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@infra-2:~#uname -as
Linux infra-2.obbana.ru 5.4.0-128-generic #144-Ubuntu SMP Tue Sep 20 11:00:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

@troman29
Copy link

ethtool -K ens3 tx off It doesn't help me either, I tried for all nodes and interfaces :(

@troman29
Copy link

I think the problem is that I have a VPS server, and the traffic just doesn't reach the client OS. I will look for other solutions

@frank-lar
Copy link

I was trying to setup a Swarm over a Hetzner private network (using a vSwitch).

Mesh routing not working, I could only make it work in global / host mode. Tried anything with the firewall and the ethtool workarounds listed above, tried to change linux distro (Almalinux 8 and Debian 11), had zero luck.

Then I found this comment on Reddit which quite saved my life.

So, if you can't get Swarm working over a Hetzner network and you already tried everything, check your MTUs: you need to adjust Docker networks MTU so that it's lower or equal 1450, which is Hetzner vlan MTU.

@ivan-spasov
Copy link

Hello everyone,

I was having the exact same issue for a swarm cluster, buit of Ubuntu 20.04.4 LTS VMs, running on ESXi 6.7. I spent countless hours troubleshooting it. My main focus was iptables, since it made most sense to me.

However, in my case, running the command below on all cluster nodes immediately fixed my problem. Now, ingress publishing works like a charm!

sudo ethtool -K <interface> tx-checksum-ip-generic off

It's worth trying!

Best regards,
Ivan Spasov

@jsalgado78
Copy link

Same issue for a swarm cluster, built of RHEL 8.6 VMs, running on ESXi 7.0.3
This issue should be resolved with RHEL 8.5 kernel 4.18.0-348 and ESXi 7.0.3 but it doesn't work. It only works with this command: ethtool -K tx-checksum-ip-generic off

https://access.redhat.com/solutions/5881451

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
20.10.x bugs/regressions
  
Needs triage
Development

No branches or pull requests