Firewalld zones interface removal is not permanent

[jon4hz]

Description
It seems to me like dockers removal of firewalld zones interfaces is not permanent.

Steps to reproduce the issue:

1. Check the current interfaces:
   # firewall-cmd --list-all --zone=docker
2. Create a new network:
   # docker network create test
3. A new bridge should be added to the interfaces:
   # firewall-cmd --list-all --zone=docker
4. Remove the network:
   # docker network rm test
5. The network should be removed here, too:
   # firewall-cmd --list-all --zone=docker
6. Reload firewalld:
   # firewall-cmd --reload
7. The previously deleted interface appears again:
   # firewall-cmd --list-all --zone=docker

Only after rebooting the server the interface stopped appearing. Neither restarting docker or firewalld could solve the issue.

Describe the results you received:
After reloading firewalld, all previously deleted docker bridges appear in the firewalld docker zone interface list.

root@server# firewall-cmd --list-all --zone=docker
docker (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: br-4be931ba093c docker0
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	
root@server# docker network create test
b3f9a7383f13b7ea814946d88e5251f54a6a05a2e875e20a89908a2c381bf072
root@server# firewall-cmd --list-all --zone=docker
docker (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: br-4be931ba093c br-b3f9a7383f13 docker0
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	
root@server# docker network rm test
test
root@server# firewall-cmd --list-all --zone=docker
docker (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: br-4be931ba093c docker0
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	
root@server# firewall-cmd --reload
success
root@server# firewall-cmd --list-all --zone=docker
docker (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: br-4be931ba093c br-b3f9a7383f13 docker0
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	
root@server# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:be:11:3f brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:87:1b:5e brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:a7:79:0d:cd brd ff:ff:ff:ff:ff:ff
5: br-4be931ba093c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:03:6d:68:2b brd ff:ff:ff:ff:ff:ff
7: veth7b13b40@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4be931ba093c state UP mode DEFAULT group default 
    link/ether 0e:57:aa:b2:f4:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Describe the results you expected:
I expected that the interfaces were removed permanently and don't appear after a firewall-cmd --reload

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:42 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

lient:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-105-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.89GiB
 Name: funny-server-name
 ID: CU4J:VST2:CHFF:LJSG:7XHL:MWOR:7HKN:V5BP:JLDC:LZOK:K25K:QVAO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
The server is an ubuntu 20.04 vm hosted on an openstack cluster.

[jandsu]

Great description! :)

While this may seem harmless, it causes unwanted NAT hairpinning inside Docker networks due to duplicate rules in iptables

-A POSTROUTING -s 172.18.0.0/16 ! -o br-4a99e748fcc1 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-9dbbf26e610f -j MASQUERADE

See https://serverfault.com/a/1112975/451782

[jandsu]

My workaround for the aformentioned issue was to write a small script to prune "zombie" interfaces... It looks like:

systemctl stop docker

for interface in $(firewall-cmd --zone=docker --list-interfaces)
do
    if ! ip link ls "${interface}" >/dev/null 2>&1
    then
        firewall-cmd --zone=docker --remove-interface="${interface}"
        firewall-cmd --runtime-to-permanent
        firewall-cmd --reload
    fi
done

systemctl start docker

Notice how I had to stop the Docker daemon, because it seems to keep track of the "zombie" interface and it would re-add the NIC to the "docker" zone as soon as you reload firewalld (firewall-cmd --reload) otherwise!

To me, this demonstrate that the bug is in Docker, not in firewalld.

Reproduced with last week's Docker version 20.10.19

[polarathene]

Not sure who to ping for triage... @corhere ?

---

Confirming that this is still reproducible with current releases:

• firewall-cmd --reload will bring back the bridge iptables rules of removed interfaces.
• I did not see the networks overlap in subnet range (I used docker network create test && docker network rm test several times before reload).
• systemctl restart docker && firewalld-cmd --reload works without requiring a reboot. Docker needs to restart, but the iptables rules remained present, as did the interfaces in the docker zone until reloading firewalld. ip addr did not list the interfaces prior to invoking a reload.

---

Journalctl log output

Some logs from journalctl after systemctl restart docker:

dockerd[18066]: time="2023-05-26T06:20:41.068595770Z" level=info msg="Firewalld: docker zone already exists, returning"

firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables v1.8.8 (nf_tables):  CHAIN_DEL failed (Device or resource busy): chain DOCKER

firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
dockerd[18066]: time="2023-05-26T06:20:41.187834855Z" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"

firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-6b9813412b5d -o br-6b9813412b5d -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
dockerd[18066]: time="2023-05-26T06:20:41.258989909Z" level=info msg="Firewalld: interface br-6b9813412b5d already part of docker zone, returning"

Adding a network:

NetworkManager[804]: <info>  [1685082511.7251] manager: (br-4859a07dc6ad): new Bridge device (/org/freedesktop/NetworkManager/Devices/80)
firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-4859a07dc6ad -o br-4859a07dc6ad -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
dockerd[18066]: time="2023-05-26T06:28:31.818085115Z" level=info msg="Firewalld: interface br-4859a07dc6ad already part of docker zone, returning"

No logs for removing a network.

Invoking a reload, logs are very similar to restarting the docker service, removed interface is logged:

firewalld[9742]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-4859a07dc6ad -o br-4859a07dc6ad -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
dockerd[18066]: time="2023-05-26T06:32:26.442215962Z" level=info msg="Firewalld: interface br-4859a07dc6ad already part of docker zone, returning"

Interface is present in iptables -t nat -nvL POSTROUTING, but not docker network ls or ip addr outputs.

Restart docker service, logs like shown originally, with existing networks but not the removed network. Firewalld still outputs the zone with the removed interface until invoking reload.

---

Click to view

Client: Docker Engine - Community
 Version:           24.0.1
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        6802122
 Built:             Fri May 19 18:07:52 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.1
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       463850e
  Built:            Fri May 19 18:06:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Click to view

Client: Docker Engine - Community
 Version:    24.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.12-200.fc37.x86_64
 Operating System: Fedora Linux 37 (Server Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 947.4MiB
 Name: vpc-fedora
 ID: 91d9ebe9-0988-4d55-9030-e7cff48f5dd2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[thaJeztah]

Perhaps @akerouanton, who's making his way through networking-related tickets an creating an inventory