Stricter PIDfile handling breaks all non-root services running in docker

[voroniys]

systemd version the issue has been seen with
systemd 237

Used distribution
ubuntu-18.04

Expected behaviour you didn't see
services that create a pid file as non-root which run fine when in VM should also run executed in docker container

Unexpected behaviour you saw
Service, that runs fine in normal VM (let say opendkim) failing to start when executed in docker container:

● opendkim.service - DomainKeys Identified Mail (DKIM) Milter
   Loaded: loaded (/etc/systemd/system/opendkim.service; enabled; vendor preset: enabled)
   Active: activating (start) since Mon 2019-02-18 15:03:54 UTC; 34s ago
     Docs: man:opendkim(8)
           man:opendkim.conf(5)
           man:opendkim-genkey(8)
           man:opendkim-genzone(8)
           man:opendkim-testadsp(8)
           man:opendkim-testkey
           http://www.opendkim.org/docs.html
  Process: 750 ExecStart=/usr/sbin/opendkim -x /etc/opendkim/opendkim.conf -P /var/run/opendkim/opendkim.pid (code=exited, status=0/SUCCESS)
    Tasks: 6 (limit: 4669)
   CGroup: /docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service
           └─751 /usr/sbin/opendkim -x /etc/opendkim/opendkim.conf -P /var/run/opendkim/opendkim.pid
Feb 18 15:03:54 d3d81786f3e1 systemd[1]: Starting DomainKeys Identified Mail (DKIM) Milter...
Feb 18 15:03:54 d3d81786f3e1 opendkim[751]: OpenDKIM Filter v2.11.0 starting (args: -x /etc/opendkim/opendkim.conf -P /var/run/opendkim/opendkim.pid)
Feb 18 15:03:54 d3d81786f3e1 systemd[1]: opendkim.service: New main PID 751 does not belong to service, and PID file is not owned by root. Refusing.

Steps to reproduce the problem

• Run ubuntu-18.04 in a docker container
• log in to the container
• execute apt -y install opendkim
• execute systemctl start opendkim
• execute systemctl status opendkim to see the errors like above.

[poettering]

what does cat /proc/751/cgroup say in the case above? (i.e. what cgroup memberships does the kernel report for the new main PID 751 of the service?)

my educated guess is that docker doesn't set up cgroups correctly and /sys/fs/cgroup and /proc/$PID/cgroup do not match.

[voroniys]

Looks like this is not the case:

● opendkim.service - DomainKeys Identified Mail (DKIM) Milter
   Loaded: loaded (/etc/systemd/system/opendkim.service; enabled; vendor preset: enabled)
   Active: activating (start) since Mon 2019-02-18 16:28:00 UTC; 42s ago
     Docs: man:opendkim(8)
           man:opendkim.conf(5)
           man:opendkim-genkey(8)
           man:opendkim-genzone(8)
           man:opendkim-testadsp(8)
           man:opendkim-testkey
           http://www.opendkim.org/docs.html
  Process: 1250 ExecStart=/usr/sbin/opendkim -x /etc/opendkim/opendkim.conf -P /var/run/opendkim/opendkim.pid (code=exited, status=0/SUCCESS)
    Tasks: 6 (limit: 4669)
   CGroup: /docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service
           └─1251 /usr/sbin/opendkim -x /etc/opendkim/opendkim.conf -P /var/run/opendkim/opendkim.pid

Feb 18 16:28:00 d3d81786f3e1 systemd[1]: Starting DomainKeys Identified Mail (DKIM) Milter...
Feb 18 16:28:00 d3d81786f3e1 systemd[1]: opendkim.service: New main PID 1251 does not belong to service, and PID file is not owned by ro
ot. Refusing.

and /proc/1251/cgroup reports:

11:perf_event:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
10:cpuset:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
9:freezer:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
8:net_cls,net_prio:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
7:pids:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service
6:devices:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service
5:cpu,cpuacct:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
4:hugetlb:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
3:blkio:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
2:memory:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a
1:name=systemd:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service

and in turn

root@d3d81786f3e1:~# cat /sys/fs/cgroup/systemd//docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service/cgroup.procs
1251
root@d3d81786f3e1:~#

so they do match

[poettering]

No, this is broken:

1:name=systemd:/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/docker/d3d81786f3e154c7c9d379bcc3d4bdb1c55a5ee824195be91b8a40e55fdebe3a/system.slice/opendkim.service

it's the one line that matters... where does the duplicate hex thing comes from?

Not sure what is going on there, but this almost definitely borkage in docker how they set up cgroupfs and cgroup ns.

See: https://systemd.io/CGROUP_DELEGATION

If docker doesnt' implement the above it's always going to remain a frickin' mess...

[voroniys]

Issues trucking on GitHub is switched off for Docker, so I have no idea how to report this back to docker.

[voroniys]

Reported it to Moby as well: moby/moby#38749.
I agree this is a Docker issue, but may be workaround for this issue in systemd is also good idea.

[poettering]

Sorry, but systemd is not the place to add work-arounds for borkages in docker... We have to make certain requirements on the environment we run in, and this is well documented, and not trivial nor obvious how to work around this at all...

Let's close this here, this needs to be fixed in docker. Sorry.