rootless: First call via socket activation always times out

[WhyNotHugo]

Description

With docker rootless running via systemd socket activation, the first call to docker always times out.

The service is started fine though, but never responds to this request. Subsequent requests work fine.

➜ cat /usr/lib/systemd/user/docker.socket
[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=%t/docker.sock
SocketMode=0660

[Install]
WantedBy=sockets.target
➜ echo $DOCKER_HOST
unix:///run/user/1000/docker.sock

Manually starting docker (systemctl --user start docker.service) works fine.
The issue does not happen when running docker via the system systemd.

Steps to reproduce the issue:

1. Make sure you have the above docker.socket and docker is not running.
2. docker ps

Describe the results you received:

Docker should respond to this request.

Describe the results you expected:

Docker times out for this request.

Additional information you deem important (e.g. issue happens only occasionally):

system/docker.service has ExecStart=/usr/bin/dockerd -H fd://, whereas user/docker.service has ExecStart=/usr/bin/dockerd-rootless.sh.

Since dockerd-rootless.sh passes all arguments to docker, I tried changing that last line to ExecStart=/usr/bin/dockerd-rootless.sh -H fd://. This did not work:

Feb 25 14:16:54 victory dockerd-rootless.sh[236235]: + exec dockerd -H fd://
Feb 25 14:16:54 victory dockerd-rootless.sh[236235]: time="2021-02-25T14:16:54.139177651+01:00" level=info msg="Starting up"
Feb 25 14:16:54 victory dockerd-rootless.sh[236235]: time="2021-02-25T14:16:54.139232906+01:00" level=warning msg="Running in rootless mode. This mode has feature limitations."
Feb 25 14:16:54 victory dockerd-rootless.sh[236235]: time="2021-02-25T14:16:54.139239818+01:00" level=info msg="Running with RootlessKit integration"
Feb 25 14:16:54 victory dockerd-rootless.sh[236235]: failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Feb 25 14:16:54 victory dockerd-rootless.sh[236207]: [rootlesskit:child ] error: command [/usr/bin/dockerd-rootless.sh -H fd://] exited: exit status 1
Feb 25 14:16:54 victory dockerd-rootless.sh[236194]: [rootlesskit:parent] error: child exited: exit status 1
Feb 25 14:16:54 victory systemd[597]: docker.service: Main process exited, code=exited, status=1/FAILURE

It seems that something else needs to be done for the wrapper to pass down the file handler.

Output of docker version:

➜ docker version
Client:
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.16
 Git commit:        48d30b5b32
 Built:             Sun Feb 21 15:36:21 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16
  Git commit:       46229ca1d8
  Built:            Sun Feb 21 15:35:53 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b.m
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 34
  Running: 2
  Paused: 0
  Stopped: 32
 Images: 269
 Server Version: 20.10.3
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.11.1-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.34GiB
 Name: victory
 ID: OX7Y:NSIV:SHJC:WKRS:6MDW:ON2S:FUXA:WT6W:PTFU:P65R:W7YG:FHMP
 Docker Root Dir: /home/hugo/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

On ArchLinux. I maintain the docker-rootless-extras-bin package, which installs the rootless extras. All files are identical to what upstream bundles, with the exception of docker.socket, since upstream has no systemd.socket file.

Using pure upstream + docker.socket has the same issues.

[thaJeztah]

/cc @AkihiroSuda

[AkihiroSuda]

This is not supported in the official installation script, so let me set kind/enhancement label for this.

I'm not sure whether we can support this, but probably we can, by modifying RootlessKit to propagate FDs.

[WhyNotHugo]

This is not supported in the official installation script, so let me set kind/enhancement label for this.

Installing all the pieces manually without the script yields the same results -- or do you mean rootless in general is not officially supported?

[AkihiroSuda]

Rootless is official, but rootless with -H fd:// is not

[WhyNotHugo]

Ah, understood.

FWIW, without support for passing the fd, it would seem that manually starting docker is the only option (or what I do, which is just run the first command twice).

[firesock]

As a workaround, until rootlesskit is updated, it's possible to make it work with https://www.freedesktop.org/software/systemd/man/systemd-socket-proxyd.html, which requires an extra socket. The details get a bit fiddly, but it does work

[WhyNotHugo]

Using systmed-socket-proxyd seems doable. That would require:

• Changing the default docker.service to listen in another location (e.g.: unix:///run/user/1000/docker-inner.sock).
• Change docker.socket to run a new service instead of the current one (e.g.: docker-proxy.service). This still owns the "real" socket (unix:///run/user/1000/docker.sock).
• docker-proxy.service has Exec=systemd-socket-proxyd unix:///run/user/1000/docker-inner.sock.

The only downside I can see is that people running something like systemctl --user start docker.service also need to manually start docker-proxy.service for docker commands to work as expected. I think for most cases though, people will just enable the socket which will result on the service being started when needed, but it's still a caveat to keep in mind.

[firesock]

An ExecStartPre=/usr/bin/timeout 10 sh -c 'while ! [ -S %t/docker-inner.sock ]; do sleep 0.5s; done' in docker-proxy.service is also helpful otherwise the initial CLI command can get an EOF while docker.service is starting up