change spec file to use munge.service and tmpfiles.d for systemd

[GoogleCodeExporter]

What steps will reproduce the problem?

Boot a SLES 12 system with the munge service file enabled.

What is the expected output? What do you see instead?

Expected output is the munge service starting successfully.

What we see is munge fails to come up with

Oct 22 16:18:46 nid00002 munged[10601]: munged: Error: Failed to check pidfile dir "/var/run/munge": cannot canonicalize "/var/run/munge": No such file or directory

What version of the software are you using? On what operating system?

Munge 0.5.11 on SLES 12.

Please provide any additional information below.

In our environment systemd removes /var/run/munge before starting services, so it's missing when munged tries to start. Using this as a guide:

https://blog.hqcodeshop.fi/archives/93-Handling-varrun-with-systemd.html

I added a few lines to munge.service to fix the problem.

PermissionsStartOnly=true
ExecStartPre=-/usr/bin/mkdir -m 0755 -p /var/run/munge
ExecStartPre=-/usr/bin/chown -R munge:munge /var/run/munge

Original issue reported on code.google.com by dcgloe on 23 Oct 2014 at 8:23

[GoogleCodeExporter]

Why is systemd removing /var/run/munge? I presume this is something different than having /var/run mounted as tmpfs. I thought the general consensus for creating this directory was to use tmpfiles.d:

https://en.opensuse.org/openSUSE:Systemd_packaging_guidelines#Creating_files_.2F_subdirectories_in_.2Fvar.2Frun_and_.2Frun

https://fedoraproject.org/wiki/Packaging:Tmpfiles.d

http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

I don't have access to a SLES 12 system. Would that be comparable to openSUSE 12.x?

Original comment by chris.m.dunlap on 24 Oct 2014 at 12:33

[GoogleCodeExporter]

Before this I was trying to create the directory with ansible run before systemd when the image is booted. Perhaps systemd can somehow tell that directory isn't in the image so it's removing it to have a clean state for this boot.

I didn't know about tmpfiles.d, it looks like it would work better than my solution.

SLES 12 is still in beta, I heard it's more comparable to openSUSE 13.

Original comment by dcgloe on 24 Oct 2014 at 1:54

[GoogleCodeExporter]

I tried installing the tmpfiles.d configuration file and it works fine. I guess my only request would be to add the munge.service and tmpfiles.d file to the spec file.

Original comment by dcgloe on 24 Oct 2014 at 5:31

[GoogleCodeExporter]

In 0.5.11, I tried installing munge.service & tmpfiles.d where appropriate but my attempts at autodetection caused other problems (cf. #27, #33, #34). I'm planning to make systemd the default in the next release. Most distributions now have chosen that as the path forward. I'm updating the issue summary to reflect this. I'm hoping to have the same spec file work on Red Hat and SUSE based distros.

Original comment by chris.m.dunlap on 25 Oct 2014 at 6:31

• Changed title: change spec file to use munge.service and tmpfiles.d for systemd
• Changed state: Accepted
• Added labels: Milestone-0.5.12

[dun]

The systemd unit file and tmpfiles.d configuration file were added to the top-level specfile in commit d034b3a. But the specfile likely needs changes to work on SLES12.

The top-level munge.spec will target the latest CentOS / RHEL release. RPM specfiles for other distros & releases will be placed in the top-level extra directory -- please submit a pull request.

[kkm000]

Hi @dun, the way to support this is to specify

[Service]
RuntimeDirectory=munge

According to systemd.exec(8),

RuntimeDirectory= [...]
[...] take[s] a whitespace-separated list of directory names. The specified directory names must be relative, and may not include "." or "..". If set, one or more directories by the specified names will be created (including their parents) below /run [...] when the unit is started.
[...] the innermost specified directories will be owned the user and group specified in User= and Group=. If the specified directories already and their owning user or group do not match the configured ones, all files and below the specified directories as well as the directories themselves will have file ownership recursively changed to match what is configured.
Use RuntimeDirectory= to manage one or more runtime directories for the unit and bind their lifetime to the daemon runtime. This is particularly useful for unprivileged daemons that cannot create runtime directories in /run due to lack of privileges, and to make sure the runtime directory is cleaned up automatically after use.

This way, the tmpfiles.d file, currently packaged with the rpm, will not be needed.

On the same point, StateDirectory=munge would ensure the /var/lib/munge directory is created and has its permissions corrected before starting the daemon.

An added, albeit probably a minor benefit here is that the unit automatically gets dependencies on filesystem mounts, i. e. won't start until respective /run and /var/lib are indeed mounted:

Units with WorkingDirectory=, RootDirectory=, RootImage=, RuntimeDirectory=, StateDirectory=, LogsDirectory= or ConfigurationDirectory= set automatically gain dependencies type Requires= and After= on all mount units required to access the specified paths.

---

It may also be a good idea to add the start prerequisite condition

[Unit]
AssertFileExists=/etc/munge/munge.key

[dun]

@kkm000: RuntimeDirectory support was added recently in 3eed37e.

I don't see much benefit with StateDirectory since that directory will be created by the package, and its permissions will be checked at runtime by munged. Also, that option was added in systemd 235, and I want systemd support to be compatible with the current CentOS (currently systemd 219 on CentOS 7.6.1810). But it would be useful with DynamicUser, so I'll look at it again in the future.

I presume you mean AssertPathExists. That looks like it could be useful. I'll do some testing with it. Thanks for the tip!

[kkm000]

@dun, thank you for the tip, I did not know the StateDirectory was a relatively recent addition.

Oh, you reminded me how I had to drop Centos 7 as the platform for an internet-facing Google Cloud VM, because of a bug in systemd 219--and they support only the latest version and the one before that (and the bug I hit was reported, but was apparently in dbus, not even systemd itself). And it was the only base image with both secure boot and SELinux out of the box. I found a backport of a more recent systemd and dbus, but it required setting SELinux to permissive. Bummer!

systemd is amazingly powerful, but their EOL cycle is shamefully short.

I presume you mean AssertPathExists.

Oops, of course! I noticed there was a function in a Debian-packaged init.rc script that did exactly this, although it's not apparently in the current source. I cannot claim any credit for the idea. :)

[dun]

So I've looked into adding AssertPathExists to munge.service...

Without it, attempting to start the service before first creating a key results in systemd moving the service into a failed state -- as desired. The output from systemctl status simply shows it failed to start. The error message from munged can by obtained from journalctl:

# systemctl start munge.service
Job for munge.service failed because the control process exited with error code. See "systemctl status munge.service" and "journalctl -xe" for details.

# systemctl status -l munge.service
* munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-08-27 15:35:49 PDT; 23s ago
     Docs: man:munged(8)
  Process: 31570 ExecStart=/usr/sbin/munged (code=exited, status=1/FAILURE)

Aug 27 15:35:49 centos7 systemd[1]: Starting MUNGE authentication service...
Aug 27 15:35:49 centos7 systemd[1]: munge.service: control process exited, code=exited status=1
Aug 27 15:35:49 centos7 systemd[1]: Failed to start MUNGE authentication service.
Aug 27 15:35:49 centos7 systemd[1]: Unit munge.service entered failed state.
Aug 27 15:35:49 centos7 systemd[1]: munge.service failed.

# journalctl -xe
...
-- Unit munge.service has begun starting up.
Aug 27 15:35:49 centos7 systemd[1]: munge.service: control process exited, code=exited status=1
Aug 27 15:35:49 centos7 munged[31570]: munged: Error: Failed to check keyfile "/etc/munge/munge.key": No such file or directory
Aug 27 15:35:49 centos7 systemd[1]: Failed to start MUNGE authentication service.
-- Subject: Unit munge.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit munge.service has failed.
...

# journalctl _UID=$(id -u munge)
-- Logs begin at Fri 2019-08-16 12:34:01 PDT, end at Tue 2019-08-27 15:39:41 PDT. --
Aug 27 15:35:49 centos7 munged[31570]: munged: Error: Failed to check keyfile "/etc/munge/munge.key": No such file or directory

Adding AssertPathExists=/etc/munge/munge.key to the [Unit] section of munge.service results in the systemctl status output showing the failed assertion. No attempt is made to start munged since the assertion failed:

# vim /usr/lib/systemd/system/munge.service

# systemctl daemon-reload

# systemctl start munge.service
Assertion failed on job for munge.service.

# systemctl status -l munge.service
* munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-08-27 15:35:49 PDT; 4min 0s ago
   Assert: start assertion failed at Tue 2019-08-27 15:39:41 PDT; 8s ago
           AssertPathExists=/etc/munge/munge.key was not met
     Docs: man:munged(8)

Aug 27 15:35:49 centos7 systemd[1]: Starting MUNGE authentication service...
Aug 27 15:35:49 centos7 systemd[1]: munge.service: control process exited, code=exited status=1
Aug 27 15:35:49 centos7 systemd[1]: Failed to start MUNGE authentication service.
Aug 27 15:35:49 centos7 systemd[1]: Unit munge.service entered failed state.
Aug 27 15:35:49 centos7 systemd[1]: munge.service failed.
Aug 27 15:39:41 centos7 systemd[1]: Assertion failed for MUNGE authentication service.

While seeing the failed assertion in the systemctl status output is convenient, the munged error message is likely more informative. Furthermore, the daemon can fail to start for many reasons, and AssertPathExists only checks for the presence of the specified path. So while it's interesting, I think it's better to have munged return the specific error. But thanks for the suggestion!

[kkm000]

Amen to that! Totally convincing, thanks for investigating.