Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker swarm doesn't add some ports to iptables randomly #45171

Open
ethaniel opened this issue Mar 16, 2023 · 0 comments
Open

docker swarm doesn't add some ports to iptables randomly #45171

ethaniel opened this issue Mar 16, 2023 · 0 comments
Labels
area/networking area/swarm kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage version/20.10

Comments

@ethaniel
Copy link

ethaniel commented Mar 16, 2023 •
edited

Description

This only happens on rare events when a node gets disconnected from the swarm (most likely due to an internet hiccup, or home ISP connection drop) and then reconnected again.

I have a pihole cointainer with ports 53, 80, 67 which become inaccessible after such a reconnection due to these ports not being added back to iptables by swarm. They are, however, still listed in docker ps.

Here is the sudo docker ps output after such a reconnect (the containers were recreated by the swarm on this node 2 hours ago):

pi@home:~ $ sudo docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                 PORTS                            NAMES
e60b98152a44   ethan1el/wipe_varlog:latest   "crond -f"               2 hours ago    Up 2 hours                                              wipe_varlog.kgtl6egcsiz3xjhef38mjlwpm.nriyf2odsqfa084tnouyhqy5a
9472d775427d   crazymax/cloudflared:latest   "/usr/local/bin/clou…"   2 hours ago    Up 2 hours (healthy)   5053/udp, 49312/tcp              cloudflared.kgtl6egcsiz3xjhef38mjlwpm.jnnvhraleqfp79wyeg81ezspk
b695da0b1ff0   pihole/pihole:latest          "/s6-init"               2 hours ago    Up 2 hours (healthy)   53/udp, 53/tcp, 80/tcp, 67/udp   pihole.kgtl6egcsiz3xjhef38mjlwpm.edimgoy17rsee68px1itr7s1s
d16f4af50e88   ethan1el/upnp:latest          "crond -f"               2 hours ago    Up 2 hours                                              upnp_22_50222.kgtl6egcsiz3xjhef38mjlwpm.mwt6r8n02hmlnabrk0r6wxbp7

And here is the iptables output (please note that only the ports for cloudflared container 5053 are listed, and not the pihole ports 53, 80):

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5053
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED udp spt:5053
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Then I do service docker restart to fix everything.

Here is the new sudo docker ps (note that 53/tcp became 0.0.0.0:53->53/tcp on the pihole container, strange, but might be a hint):

pi@home:~ $ sudo docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED          STATUS                        PORTS                                                                NAMES
6d849a963bec   ethan1el/wipe_varlog:latest   "crond -f"               36 seconds ago   Up 34 seconds                                                                                      wipe_varlog.kgtl6egcsiz3xjhef38mjlwpm.5j7kaj7ojuyhjwf65vn7b27f3
86858d889e7e   crazymax/cloudflared:latest   "/usr/local/bin/clou…"   36 seconds ago   Up 32 seconds (healthy)       5053/udp, 49312/tcp                                                  cloudflared.kgtl6egcsiz3xjhef38mjlwpm.zynrspb16affoa954wrapwab6
a1aa39050f7f   pihole/pihole:latest          "/s6-init"               40 seconds ago   Up 35 seconds (healthy)       0.0.0.0:53->53/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:53->53/udp, 67/udp   pihole.kgtl6egcsiz3xjhef38mjlwpm.m45q04yo92k8kz57mxstipbld
07b33f543c18   ethan1el/upnp:latest          "crond -f"               46 seconds ago   Up 42 seconds                                                                                      upnp_22_50222.kgtl6egcsiz3xjhef38mjlwpm.xtn4jzzw41urf4oe3eqktdynz

And here are the new (correct) iptables (please note that the DOCKER chain is now filled correctly with pihole ports 53, 80):

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            172.17.0.2           udp dpt:53

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5053
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED udp spt:5053
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Reproduce

I don't have the exact steps to reproduce it as it happens only at night when I sleep (and my home internet connection gets reset).

Expected behavior

Swarm should correctly add all the ports back to iptables, so they would be accessible from outside.

docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:27:56 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:44 2022
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)

Server:
 Containers: 8
  Running: 5
  Paused: 0
  Stopped: 3
 Images: 98
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: kgtl6egcsiz3xjhef38mjlwpm
  Is Manager: true
  ClusterID: d018y2dhw6vaxf00qbzukzd5d
  Managers: 6
  Nodes: 7
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 20 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.10.13.20
  Manager Addresses:
   10.10.13.116:2377
   10.10.13.11:2377
   10.10.13.12:2377
   10.10.13.140:2377
   10.10.13.20:2377
   10.10.13.22:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.103-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 3.749GiB
 Name: home
 ID: 5O7N:JX4B:FBYE:UVSQ:N6HV:FZX3:5R6N:X77H:GACA:WZOW:YBF4:YFK5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

Additional Info

The pihole services was created via:

sudo docker service create \
  --name pihole \
  --mode global \
  --publish published=80,target=80,mode=host,protocol=tcp \
  --publish published=53,target=53,mode=host,protocol=tcp \
  --publish published=53,target=53,mode=host,protocol=udp \
  -e TZ=Asia/Bangkok \
  -e WEBPASSWORD=admin \
  --mount type=volume,src=pihole_app,dst=/etc/pihole \
  --mount type=volume,src=dns_config,dst=/etc/dnsmasq.d \
  --log-driver journald \
  --with-registry-auth \
  --no-resolve-image \
  --constraint node.labels.home==1 \
  pihole/pihole:latest
@ethaniel ethaniel added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage labels Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking area/swarm kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage version/20.10
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ethaniel @thaJeztah