permission problems: used to work

[GoogleCodeExporter]

I had munge working, basic tests across my machines worked. For some reason, it doesn't work any more and complains on ownership issues of the log file. Specifically,

/var/log/munge is owned by munge and gives the following error

/etc/init.d/munge start
 * Starting MUNGE munged
   ...fail!
munged: Error: Logfile is insecure: invalid ownership of "/var/log/munge"

if I change ownership to root, I get the following ,,,

 * Starting MUNGE munged
   ...fail!
munged: Error: Failed to open logfile "/var/log/munge/munged.log": Permission denied

I am surprised as this used to work ... who should own /var/log/munge?

What version of the software are you using? On what operating system?

0.5.11 on ubuntu 14.04

Original issue reported on code.google.com by hsundar on 16 Oct 2014 at 10:13

[GoogleCodeExporter]

The /etc/munge and /var/{lib,log,run}/munge directories (and the files therein) should all be owned by the effective UID of the running munged process. This can be a non-privileged "munge" user (or any user, for that matter) or root. Since munged doesn't require root privileges on Linux, I would recommend creating a "munge" user, changing the aforementioned directories/files to be owned by that user, and running munged as that user.

The first error (Logfile is insecure: invalid ownership of "/var/log/munge") is because /var/log/munge is not owned by either root or the euid of the running munged process.

The second error (Failed to open logfile "/var/log/munge/munged.log": Permission denied) is because munged was unable to open /var/log/munge/munged.log for writing due to insufficient permissions. This makes sense if you changed ownership of the file to root and then ran munged as a non-privileged user.

On Ubuntu, munged should be running as the non-privileged "munge" user, and these directories (and files within) should be owned by that user. Please verify the following:

1. Check /etc/init.d/munge to verify the variable USER="munge" is defined.
2. Check /etc/default/munge to verify the USER variable is not redefined to something else.
3. Check /etc/passwd to make sure the munge user exists.
4. Check to make sure you're running /etc/init.d/munge start as root.

On Ubuntu 14.04, you'll need to run munged with either the --force or the --syslog option for now due to #31. These options can be specified in the OPTIONS variable in /etc/default/munge.

Original comment by chris.m.dunlap on 17 Oct 2014 at 12:18

• Added labels: Type-Support
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by chris.m.dunlap on 20 Oct 2014 at 6:28

• Changed state: Done

[eddelbuettel]

I am still running into this:

• USER is set in /etc/init.d/munge but not in /etc/default/munge:

$ grep USER /etc/init.d/munge 
unset DESC DAEMON CONFIG OPTIONS PIDFILE NICE USER SIGHUP_RELOAD
USER="munge"
#   $CONFIG, $OPTIONS, $PIDFILE, $USER, $NICE, $SIGHUP_RELOAD
  [ "`id | sed 's/^uid=\([0-9]*\).*/\1/'`" -ne 0 ] && unset USER
    [ -n "$USER" ] && chown "$USER" "$PIDDIR"
      ${USER:+"--chuid"} ${USER:+"$USER"} \
$ grep USER /etc/default/munge 
$ 

• The user exists:

$ grep munge /etc/passwd
munge:x:126:136::/home/munge:/bin/false
$ 

• All directories look fine:

$ ls -ld /etc/munge/ /var/*/munge
drwxr-xr-x 2 munge munge 4096 Jul 29 14:21 /etc/munge/
drwxr-xr-x 2 munge munge 4096 Jul 29 14:50 /var/lib/munge
drwxr-xr-x 2 munge munge 4096 Jul 29 14:49 /var/log/munge
drwxr-xr-x 2 munge munge   40 Jul 29 14:50 /var/run/munge
$ 

But no luck:

$ whoami
root
$ /etc/init.d/munge start
[....] Starting munge (via systemctl): munge.serviceJob for munge.service failed. See "systemctl status munge.service" and "journalctl -xe" for details.
 failed!
$ systemctl status munge.service
● munge.service - MUNGE authentication service
   Loaded: loaded (/lib/systemd/system/munge.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2015-07-29 15:26:37 CDT; 5s ago
     Docs: man:munged(8)
  Process: 26946 ExecStart=/usr/sbin/munged (code=exited, status=1/FAILURE)

Jul 29 15:26:37 dub systemd[1]: Starting MUNGE authentication service...
Jul 29 15:26:37 dub munged[26946]: munged: Error: Logfile is insecure: group-writable permissions set on "/var/log"
Jul 29 15:26:37 dub systemd[1]: munge.service: control process exited, code=exited status=1
Jul 29 15:26:37 dub systemd[1]: Failed to start MUNGE authentication service.
Jul 29 15:26:37 dub systemd[1]: Unit munge.service entered failed state.
Jul 29 15:26:37 dub systemd[1]: munge.service failed.
$ 

$ chmod g-w /var/log
$ /etc/init.d/munge start
[ ok ] Starting munge (via systemctl): munge.service.
$ 

I tried --force several times, no luck. Could we / should we remove the group-write tests?

Ubuntu 15.04, same munge packages as before (ie 0.5.11-1).

[dun]

Ubuntu 15.04 switched to systemd. It doesn't use /etc/init.d/munge or /etc/default/munge.

You'll need to do the following:

1. systemctl edit --system --full munge

   This command will perform the following actions:

   1. Copy munge.service to /etc/systemd/system/munge.service
   2. Invoke an editor on this new munge.service file
   3. Reload the systemd configuration afterwards

   While in the editor, append either --syslog or --force to the ExecStart line:

   ExecStart=/usr/sbin/munged --syslog
   

   If munge.service is instead manually edited, a systemctl daemon-reload will be necessary to reload the systemd configuration afterwards.

2. systemctl enable munge

   This command will enable the service at boot.

3. systemctl start munge

   This command will start the daemon.

With these steps, I'm able to start munge-0.5.11-1.1build1 on Ubuntu 15.04.

[eddelbuettel]

I'll try that tomorrow. For now I made do via chmod g-w /var/log; /etc/init.d/munge start; chmod g+w /var/log which works, but it tedious.

What about skipping the test though? If /var/log/munge/ exists, do you really need to obsess over group permissions for /var/log?

I'd love to see this squared up. I maintained slurm for a while inside Debian before handing it off to Gennaro.

[dun]

If a directory is writable by a particular user/group, that user/group is able to rename/replace directories and files residing within it. The permission check is perhaps overkill here for /var/log, but the code uses the same routine (path_is_secure()) to check ownership & permissions on all of its directories.

I'm thinking the best path forward at this point is to make --syslog the default behavior in the next release (which is long overdue, but I've been busy with other projects).

Are you aware of any distros besides Ubuntu 14.04+ that set /var/log to 0775?

[eddelbuettel]

I am not. I am still quite involved with Debian (and even maintained slurm and openmpi for a bit a looooong while ago before Gennaro took over) but I can't speak to other distros. We use Ubuntu at work,

[eddelbuettel]

No luck editing the systemd file:

$ sudo vi /lib/systemd/system/munge.service 
$ sudo service munge stop
 * Stopping MUNGE munged                                                                                                                                                                                      [ OK ] 
$ sudo service munge start
 * Starting MUNGE munged                                                                                                                                                                                      [fail] 
munged: Error: Logfile is insecure: group-writable permissions set on "/var/log"
$ grep force /lib/systemd/system/munge.service
ExecStart=/usr/sbin/munged --force
$ sudo su -c "chmod g-w /var/log; service munge start; chmod g+w /var/log"
 * Starting MUNGE munged                                                                                                                                                                                      [ OK ] 
$ 

Can you share more details as to what you edited? Do I need need quotes, or Python-style [ ] here?

[dun]

I think the problem you're having is that you didn't reload the systemd manager configuration (systemctl daemon-reload); as such, the changes you made to munge.service didn't take effect. With systemd, I think you'll have fewer problems if you use systemctl directly instead of using the service wrapper provided for backward compatibility.

But my directions above apparently aren't best practice. You shouldn't be editing service files in /lib/systemd/system/ since package updates may overwrite those; instead, you should copy the service file to /etc/systemd/system/ before making changes there. Or you can invoke systemctl edit --system --full munge which will: 1) copy the service file to /etc/systemd/system/, 2) open an editor, and 3) reload the systemd configuration afterwards. I'll update my comment above to reflect this.

You don't need quotes or Python-style [ ]. Just append either --syslog or --force to the ExecStart option. After editing, my updated munge.service file is as follows:

$ cat /etc/systemd/system/munge.service
[Unit]
Description=MUNGE authentication service
Documentation=man:munged(8)
After=network.target
After=time-sync.target

[Service]
Type=forking
ExecStart=/usr/sbin/munged --syslog
PIDFile=/var/run/munge/munged.pid
User=munge
Group=munge
Restart=on-abort

[Install]
WantedBy=multi-user.target

Afterwards, you'll need to invoke systemctl enable munge to enable the service at boot, and invoke systemctl start munge to start the daemon now.

[eddelbuettel]

Thanks for the systemd tutorial.

(I don;t think I need to enable the service at boot. It seems to carry things over from the init.d-based setup.)

[eddelbuettel]

And I guess we should report all this to the Debian maintainer too...

[dun]

/var/log is 0755 in Debian 8 (Jessie), so --syslog or --force aren't needed there. This only seems to impact Ubuntu 14.04+.

[eddelbuettel]

Noted, thanks for the clarification. That may make it harder to get the required fix in. I know (at least a few of) my fellow Debian co-maintainers but it is a little less clear there as they don't have the 'person (or team) to package' mapping.

[eddelbuettel]

I copied your /etc/systemd/system/munge.service and tried restarting the service -- but with either one of --force or --syslog I still get the same /var/log complaint.

I think I may just bail and (locally) patch the sources and disable the test.

[dun]

Have you tried starting it from the command-line with either --force or --syslog?

[eddelbuettel]

Yes, that does indeed work. And I then got very confused about why adding to /etc/default/munge did not work -- but you later cleared that up as the file is more or less ignored now.

[dun]

It looks like you're not getting systemd to pass the modified command-line options to the invocation of munged. The steps I listed above worked for me on a pristine 15.04 VM. Very strange.

[eddelbuettel]

Indeed. And I have a pristine new machine with 15.04. I may just do an apt-get source munge one of these days and try poke around to echo argv. This ought to be debug-able, and I'd love for us to come back with something for Gennaro to put into the package so that it work out of the box on Debian and Ubuntu.

[eddelbuettel]

Good news, and I stand corrected. I must have been sloppy earlier when testing the systemd config script. It works now -- once I do sudo systemctl daemon-reload which I might have skipped earlier by accident. Now with the add --force the error becomes a warning but munged starts.

[dun]

Good to hear! Thanks for following-up.

I don't believe daemon-reload is needed with a systemctl edit, but it is necessary if you manually edit the service file.

--force will override errors (turning them into warnings) for an existing local domain socket, a lack of entropy for the PRNG, and some potentially insecure file/directory permissions.

--syslog will bypass the permission check on /var/log since log messages will instead be routed to the syslog service.

[eddelbuettel]

Yes, I edited by hand (old habits die hard).

I like separate logs, hence my preference for --force over --syslog.

[eddelbuettel]

And I now patched/modified munged and built a local package for our cluster. That was easier.

[dun]

We typically use cfengine for distributing configuration files on our clusters.

[eddelbuettel]

Sure and understood -- but I really wanted different behaviour of not checking /var/log, rather than turning an error into a warning.

[Kuchiriel]

This was very usefull, thank you very much guys!

[dun]

@Kuchiriel You're welcome. Glad you found it helpful.

[evolbeginner]

I tried ExecStart=/usr/sbin/munged --force and it worked on Centos 7.3