Upgrade to 20.10 breaks swarm network

[oneumyvakin]

Description

Steps to reproduce the issue:

1. Install Docker 19.03 on Ubuntu 20 or CentOS 8
2. Init Swarm
3. Start some services by docker stack deploy
4. Upgrade docker from 19.03 to 20.10

Describe the results you received:
Containers of services can't start
there is error in logs

Dec 10 06:21:03 dockerd[3160859]: time="2020-12-10T06:21:03.150920367Z" level=error msg="fatal task error" error="starting container failed: container 9f93a21ac2e3be11a65c91f3cfde555a415eea47c636bef432d5d2e4b08afff4: endpoint create on GW Network failed: failed to create endpoint gateway_f8cabe848464 on network docker_gwbridge: network 28d599d44202f2acdc85e42437332ddb41a81bd7f0622bc0724761ec9b49082a does not exist" module=node/agent/taskmanager node.id=u7qdqny1doho69k3nariuo1ru service.id=vhtg6aoyt360k7mluiwmshqf0 task.id=aqgsfcw88m5kujnrly74o4wh4

Describe the results you expected:
containers are running

Additional information you deem important (e.g. issue happens only occasionally):
we have two installations with this issue which happened after upgrade to 20.10
recreating services didn't help
re-initing swarm didn't help

# docker network list
NETWORK ID     NAME              DRIVER    SCOPE
e45b9b63c4ae   bridge            bridge    local
28d599d44202   docker_gwbridge   bridge    local
2aa80dc0cc04   host              host      local
w9rpuika2x0d   ingress           overlay   swarm
62f0fb2fdf28   none              null      local

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:40 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: u7qdqny1doho69k3nariuo1ru
  Is Manager: true
  ClusterID: rdq2vi44m2lkz34tdow1dvip4
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 127.0.0.1
  Manager Addresses:
   127.0.0.1:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-56-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.749GiB
 Name: cloud.filesanctuary.net
 ID: JBWW:XVUE:3XW4:OQYT:HJHK:OSRV:PFHK:PFZP:S3DV:HPZ7:NYWD:OWQO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support

[thaJeztah]

Thanks for reporting; I just tried updating a test-cluster I had running on docker 19.03, but it didn't reproduce on that cluster; if you can find more details (perhaps daemon logs contain useful information?) that would be appreciated.

/cc @arkodg @dperny

[oneumyvakin]

OK, I've resolved the issue, but not sure what was the root cause.

I've tried to downgrade docker and containerd to 19.03 and 1.3.7, but it didn't helps.

I've spotted that docker starts using firewalld to add interfaces docker0 and docker_gwbridge to "docker" zone.

On our installations we've added docker_gwbridge in "trusted" zone.

To resolve the issue I've to:

• remove docker_gwbridge from "trusted" zone in firewalld
• delete "docker" zone in firewalld
• deinstall Docker and containerd
• delete folder /var/lib/docker/network/files/
• install latest version of docker and containerd

[Nowheresly]

We experienced the same issue on our swarm cluster (OS: ubuntu 16.04, no firewalld):

From the daemon log of a worker node, I can see:

dockerd[2228]: time="2020-12-14T16:38:2902148446+01:00" level="warning" msg="[resolver] connect failed: dial udp:XX.XX.XX.XX:53: connect: network is unreachable"

We don't have this error on manager nodes.

[Nowheresly]

Ok, after re-reading the issue,we don't experience the same issue. Our containers start but they randomly fail see each other (unexpected EOF, connection closed) even if they are in the same network (driver: overlay).

[oneumyvakin]

Looks like this issue actual for CentOS 8 as well.

[sgohl]

any news here? I have the same problem on a fresh CentOS8.3 install (also Stream, from netinstall)

dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf -y install docker-ce

> docker -v
Docker version 20.10.2, build 2291f61

> cat /etc/*release*
CentOS Linux release 8.3.2011

> rpm -qa | grep docker-ce
docker-ce-rootless-extras-20.10.2-3.el8.x86_64
docker-ce-cli-20.10.2-3.el8.x86_64
docker-ce-20.10.2-3.el8.x86_64

> rpm -qa | grep containerd
containerd.io-1.4.3-3.1.el8.x86_64

> uname -r
4.18.0-269.el8.x86_64

 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
  Data Path Port: 4789
  Default Address Pool: 10.0.0.0/8  
  Network: bridge host ipvlan macvlan null overlay
 Cgroup Driver: cgroupfs
 Cgroup Version: 1

after swarm init (3 managers, no workers), creating a test service

docker service create --name nginx --publish 88:80 --constraint 'node.role == manager' nginx:alpine

curl always fails on every node except the node where the replica is actually running

disabled selinux (permissive) and firewalld to eliminate as reason
have no custom daemon.json applied