No networking in rootless docker with firewalld

[liskin]

Description

When running the docker daemon rootless, it still attempts to detect and use firewalld. If it succeeds (more on that later), iptables rules for NAT (necessary for traffic to be routed out of the docker0 bridge) are set up in the host network namespace instead of the network namespace dockerd runs in, so networking doesn't work. This is what the traffic looks like on the slirp4netns tap0 in the dockerd namespace:

15:30:41.600319 IP 172.17.0.2.52323 > 10.0.2.3.53: 1825+ A? deb.debian.org. (32)
15:30:41.606028 ARP, Request who-has 172.17.0.2 tell 10.0.2.2, length 28

No reply, obviously, 172.17.0.2 is connected to the bridge, it's meant to be masqueraded when forwarded to tap0.
Running nsenter -U --preserve-credentials -n -m -t $(cat $XDG_RUNTIME_DIR/docker.pid) /usr/sbin/iptables-save gives no output whatsoever, because there are no rules inside the net namespace.

Now the important bit: this issue can only be reproduced with recent godbus/dbus (5.0.5+) because versions before that fail to connect to dbus from inside the user namespace. This is because it's uid 0 in that namespace, it tells dbus it's uid 0 (AUTH EXTERNAL 30\r\n), and from dbus' point of view it's obviously not uid 0, so it rejects the connection, and dockerd thinks there's no firewalld and correctly uses iptables as it should inside a network namespace. This auth issue is fixed in godbus/dbus 5.0.5. The 20.10 branch of moby vendors godbus/dbus 5.0.3 so it isn't affected, but moby 22.06 and master vendor godbus/dbus 5.0.6 so the bug is reproducible there. I've also reproduced the issue with Debian's packaging of moby 20.10 which doesn't use the vendored godbus/dbus and is built against godbus/dbus 5.0.6 instead (and I've reported the issue to Debian as well).

tl;dr of the above: Not reproducible with 20.10.17, reproducible with 22.06 and master as of 2022-07-10, and also affects 20.10.14 in Debian which is built against godbus/dbus 5.0.6.

What I think the fix might look like: in libnetwork/iptables, firewalld should only be used when not running rootless, as it makes no sense to set up iptables rules in the host network namespace while the bridge is in another network namespace.

Steps to reproduce the issue:

1. build recent dockerd with godbus/dbus 5.0.5 or later (22.06 or master as of 2022-07-10 has that); alternatively install docker.io from Debian testing
2. install and start firewalld
3. (make sure polkit allows your user to change firewalld settings—otherwise dockerd detects firewalld but fails later and just terminates; polkit will normally ask interactively for authorisation but won't if experimenting in a minimal VM)
4. set up dockerd rootless (using dockerd-rootless-setuptool.sh)
5. start a container (e.g. debian:testing)
6. try to access the internet (e.g. apt update)

Describe the results you received:
fails after it realises it can't resolve any hostnames

Describe the results you expected:
internet works

Additional information you deem important (e.g. issue happens only occasionally):
There's a workaround which involves bind-mounting /dev/null over /run/dbus/system_bus_socket and hoping nothing else breaks.

Output of docker version:

Client:
 Version:           20.10.14+dfsg1
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        a224086
 Built:             Sun May  1 19:59:40 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          dev
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.18.3
  Git commit:       0910306bf9
  Built:            Sun Jul 10 16:24:51 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit4121490
 slirp4netns:
  Version:          1.0.1
  GitCommit:        6a7b16babc95b6a3056b33fb45b74a6f62262dd4

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: dev
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 5.18.0-2-amd64
 Operating System: Debian GNU/Linux bookworm/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.768GiB
 Name: deb-docker3
 ID: bdd42690-abb4-494e-91d1-fad7d00297df
 Docker Root Dir: /home/vagrant/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):
Reproduced on both my laptop and in a VM (debian testing image from vagrant), using both the version of docker.io shipped with Debian and binaries freshly built from this repo today. Not reproduced when using the packages from https://download.docker.com/linux/debian as those are built with the older vendored godbus/dbus.

[liskin]

I reported this to Debian and one of the maintainers pointed out that I was looking at an outdated vendor.conf for master and that godbus/dbus has indeed already been bumped to 5.0.6 in both 22.06 branch and master. Therefore I've updated the description to say that it likely affects master and 22.06 as well. I'll try to actually reproduce this later.

[liskin]

Indeed, reproducible in moby master as well. Description updated.
(Sorry for the noise, hopefully this is my latest update. 😇)

[AkihiroSuda]

@liskin Thanks for reporting this, could you confirm this PR fixes the issue?

• libnetwork: skip firewalld management for rootless #43793