Propose better way to run docker from a unit file

[ibuildthecloud]

Systemd does a lot of stuff. Docker does a lot of stuff. That stuff may or may not overlap. I don't really care. I just need to solve one very specific problem. I just need a sane way to launch Docker containers in a systemd environment as a system service. As it stands today, the only way I know how is to do docker start -a or docker run ... without -d. Then dockerd launches the container in the background and systemd essentially monitors the docker client. Two problems with this. First, whether or not the docker client is running says very little about whether the actual container is running. Second, I'm left with a rather large docker run process in memory that's not providing much value except to stream stdout/stderr to journald.

So I hacked up the below script to make things better, or really just to see if it was possible to make things better since the script is just a dirty hack. You don't really need to read the script, just skip down and I'll explain what it does.

#!/bin/bash
set -e

ID=$(/usr/bin/docker "$@")
PID=$(docker inspect -f '{{.State.Pid}}' $ID)

declare -A SRC DEST

for line in $(grep slice /proc/$PID/cgroup); do
        IFS=: read _ NAME LOC <<< "$line"
        SRC[${NAME##name=}]=$LOC
done 

for line in $(grep slice /proc/$$/cgroup); do
        IFS=: read _ NAME LOC <<< "$line"
        DEST[${NAME##name=}]=$LOC
done

for type in ${!SRC[@]}; do
        from=/sys/fs/cgroup/${type}${SRC[$type]}
        to=/sys/fs/cgroup/$type/"${DEST[$type]}"/$(basename "${SRC[$type]}")

        echo $from "=>" $to
        mkdir -p $to
        for p in $(<$from/cgroup.procs); do
                echo $p > $to/cgroup.procs
        done
done

echo $PID > /var/run/test.pid

Then I wrote the following unit file

[Unit]
Description=My Service
After=docker.service
Requires=docker.service

[Service]
ExecStart=/opt/bin/docker-wrapper.sh run -d busybox /bin/sh -c "while true; do echo Hello World; sleep 1; done"
Type=forking
PIDFile=/var/run/test.pid

[Install]
WantedBy=multi-user.target

So what this does (and I know it's a hack, but I wanted to see if my proposal has any chance of working) is that after the container is launched, I look up the PID of the container and all of its cgroups. I then create child cgroups of the systemd cgroups and then move the PIDs from the original cgroups to the systemd child cgroups. After that is done I then write the PID of the container to a file. I end up with systemd cgroups being the parent, then a child cgroup under that. Looking something like below

  ├─test.service
  │ └─docker-8a0ff7503e0fca4f44d48f76a24cbcae82079818e3ad4d0d707ccf5765698184.scope
  │   ├─19103 /bin/sh -c while true; do echo Hello World; sleep 1; done
  │   └─19169 sleep 1

Also, since I told systemd to use a PIDFile, systemd is monitoring the PID 1 of the container because I wrote it to a file. So now if I do either docker stop or systemctl stop things just work (at least they seem to do) and I don't have a useless docker client hanging around in memory Now if you look at the script, you'll notice I'm just moving the PIDs, not the settings, so yeah, total hack that defeats the purpose of the original cgroup, but that's not the point right now.

Here's what I propose to make systemd and docker integration a tad bit better. When you want to run docker in a systemd unit you run docker run/start --yo-dawg-use-my-cgroups-as-your-parent ... which will read the current /proc/$$/cgroup of the client and pass it to dockerd. Dockerd now just creates its cgroups as a child of the cgroups passed in, if the subsystem exists. I think this means we can remove the systemd cgroup code and just use the cgroup fs based code (but docker will still have to write to the name=systemd fs). So now systemd can setup the parent cgroups however it wishes and Docker can setup the child cgroups how ever it wishes.

Is this the best solution? Probably not. But it seems a lot better than what we have today and it solves a current pain point.

Is this just plain stupid or already been thought of and shot down?

[vbatts]

From the [significant]discussion around systemd unit files in the contributor meeting yesterday https://botbot.me/freenode/docker-dev/msg/17771621/). The example unit files is @crosbymichael https://github.com/crosbymichael/.dotfiles/blob/master/systemd/redis.service

[daurnimator]

@vbatts link to @crosbymichael 's repo is now missing. Old version available at https://github.com/crosbymichael/.dotfiles/blob/9b7e4dc76c912ff62b445c16f620aeb8f48e3cf6/systemd/redis.service

[ibuildthecloud]

FYI, for anybody who stumbles upon this issue. I created https://github.com/ibuildthecloud/systemd-docker as an attempt to address the issues between docker and systemd.

[pikeas]

Any new thoughts/movement on this?

[tbatchelli]

I have been using @ibuildthecloud's systemd-docker and the combo is a killer. Would be better if the issues it addresses issues were dealt with by docker itself

[larsks]

This issue is hardly specific to systemd. It affects any environment in which someone wants to reliably start and monitor a container, which would include just about any non-SysV init system (systemd, upstart, runit, daemontools, launchd).

[ibukanov]

A simpler solution then using @ibuildthecloud's systemd-docker is to start a docker container in the background in ExecStartPre via run -d container or start container and then using ExecStart=/usr/bin/docker logs -f container. This way systemd, before starting any dependent units, waits until docker run -d or docker start returns and that happens only when the container is started. Then the logs command sends the initial startup logs to systemd and journal and then continue to do so as the new logs arrive until the container stops.

With this approach one also needs to put -/usr/bin/docker stop container both to ExecStop and ExecStopPost. The latter ensures that if /usr/bin/docker logs dies before the container terminates, then systemd still stops the container. Note that by just using ExecStopPost without ExecStop one will not get the termination logs into the journal as systemctl stop will kill the logs command before ExecStopPost stops the container.