Disabling userland proxy in rootless installation

[chaptergy]

Description

When updating the daemon.json to contain "userland-proxy": false, no network connectivity at all seems to be possible to the docker containers. Is this a bug or is this a limitiation of the rootless mode?

Steps to reproduce the issue:

1. Install Docker in rootless mode
2. Create a container with some exposed ports. (If you haven't changed the privileged ports system settings make sure this port is >1024).Connecting to them should be possible.
3. Update your ~/.config/docker/daemon.json to be

   {
       "userland-proxy": false
   }

4. Restart the docker service by running systemctl --user restart docker

Describe the results you received:
Connecting to the exposed port of the container is no longer possible. This does not change by adding "iptables": true to the daemon config json object.

Describe the results you expected:
Connecting to the container should be possible just as before.

Output of docker version

Client:
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.10
 Git commit:        dea9396
 Built:             Thu Nov 18 00:34:03 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:38:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 20.10.11
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 4.19.0-18-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.792GiB
 Name: hostkiste
 ID: JY3V:RRSO:Y36L:OGOE:ZCR5:ABHZ:3TM7:WZAP:S4PJ:UARF:DCAJ:XFFC
 Docker Root Dir: /home/dky942/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

VPS machine, not sure what technology they use for their VPSs

[AkihiroSuda]

Expected behavior

[chaptergy]

Is there just no way to disable the userland proxy when running in rootless mode, or would there be an alternative like allowing passwordless sudo for the iptables command? I'm not too familiar with the container network internals, so I'm not really sure at which point this fails, I'm just assuming it has to do with the iptables entry.

[griffinht]

https://docs.docker.com/engine/security/rootless/#networking-errors mentions "This is an expected behavior, as the daemon is namespaced inside RootlessKit’s network namespace.". Thus, I don't think it is possible to disable the userland proxy.

I'm not sure why you are looking to disable userland proxy, but if it is to propogate source IP addresses (see the real IP address of connecting external clients to your container) then see the docs under the section "docker run -p does not propagate source IP addresses" for a solution.

[chaptergy]

Thanks for the link I must have overlooked that section. The IP was one of the reasons I wanted to disable it. The other was to improve performance by removing the proxy, which I read about in some other issue, where it was also referenced, that removing the userland proxy might become default in the future within moby. However I can't find the issue anymore.

However following the documentation and creating the file does not seem to work, as it crashes the daemon / prevents it from starting and there does not seem an easy way to view the logs of the rootless daemon. Though that is off topic for this issue.

[griffinht]

I also spent a fair bit of time today to find the workaround and only found this Github issue because you had the same problem as me. I also read about performance issues with the proxy, and it looks like this method further reduces performance according to the linked benchmark.

I wish there was a way to get the same performance as rootful docker. I would consider --net=lxc-user-nic, but it is experimental and seems complicated, as mentioned here.

For me, adding the file and restarting the daemon worked perfectly the first time. Maybe this was because I am running on a freshly provisioned minimal headless Debian server. Too bad you had issues with slirp4netns. Rootless docker in general seems fairly experimental, so hopefully it will improve over time.

[AkihiroSuda]

However following the documentation and creating the file does not seem to work, as it crashes the daemon / prevents it from starting and there does not seem an easy way to view the logs of the rootless daemon. Though that is off topic for this issue.

Probably your slirp4netns is too old. Needs to be v0.4.0 or later. v1.1.x is recommended.

Logs can be fetched via journalctl --user -u docker.

Or just run dockerd-rootless.sh without systemd

systemctl --user stop docker

DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns dockerd-rootless.sh

[williamdes]

Even better, in /usr/bin/dockerd-rootless.sh it is explained.

mkdir ~/.config/systemd/user/docker.service.d/

In: ~/.config/systemd/user/docker.service.d/override.conf

[Service]
Environment=DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER="slirp4netns"

See: #42293 (comment)

But in fact I was missing: sudo modprobe br_netfilter. The command DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns dockerd-rootless.sh showed this

At the end it still not worked, so I used different ports for each container.
Instead of 127.0.0.1:8080:8080 and 127.0.0.2:8080:8080, I now have 127.0.0.1:8080:8080 and 127.0.0.2:8081:8080
this is since rootless mode