`systemctl mask foo` doesn't work when foo is a service configured "locally"

[vcaputo]

systemd version the issue has been seen with
v241

Used distribution
Arch

Expected behaviour you didn't see
service masked to prevent starting, e.g:

# systemctl mask foo
Created symlink /etc/systemd/systemctl/foo.service        /dev/null.

Unexpected behaviour you saw

# systemctl mask foo
Failed to mask unit: File /etc/systemd/system/foo.service already exists.

Steps to reproduce the problem
Configure a service in /etc/systemd/system, and try to mask it using systemctl.

One must manually delete, rename or otherwise backup the service file just because it's a locally configured service in order to mask it.

From the perspective of the user running systemctl mask foo, who may or may not have been responsible for configuring the foo service, this is completely spurious and inconsistent behavior and it's entirely undocumented in the systemctl man page for the mask command.

The scenario where I encountered this was I needed to prevent an old, locally-configured service, from getting started via systemctl. I had no intention of removing the service, messing with its configuration, or anything of the sort. It was being started and stopped by udev rules based on solar-powered AC availability, and I needed to temporarily prevent those rules from succeeding in starting the unit because the service is too power hungry and the skies are too cloudy this month (solar) to support it. So a simple systemctl mask foo is all that was desired, to be unmasked when the skies cleared at a later date.

It's my opinion that systemctl should have a dedicated top-priority directory for sticking such overriding service links, rather than sharing the /etc/systemd/system namespace with other local configuration. Mask links are arguably overrides to the global configuration, including local configuration, rather than part of the local configuration itself. That's certainly how they relate to distro-provided services in /lib/systemd/system, I see no good reason for them to be narrowly applicable to only those.

In the example above I used /etc/systemd/systemctl as such a directory, but anything reasonable would work. Maybe /etc/systemd/overrides or something would make more sense, but that's implementation detail.

My intention here is to illuminate the issue and try get us to at least agree this is a warty bug.

[arvidjaar]

Actually I would say that adding Masked=true directive support would allow masking of any unit wherever it is located by appropriate drop-in.

OTOH /etc is now misused for both locally defined units and for local changes to existing units. Originally there was not way to change existing unit except by copying it into /etc so both were equal. Now, with elaborate drop-in support, may be some other place for local unit definitions (not overrides) need to be established.

[poettering]

Well, masking a unit is done by symlinking it to /dev/null in /etc/systemd/, and that's documented. Which means you cannot really have both a file there and mask them, because you cannot have a single file be both a regular file and a symlink to /dev/null...

If we wanted to supported what you are asking for we'd have to add yet another directory we have in our search path, but I am not sure this is really helperful, as it makes things even more baroque to the user. Right now we distinguish only "local user configuration" stuff (i.e. in /etc) from "vendor" stuff (i.e. in /usr) basically. If we'd want to support the ability to mask user stuff, then we'd have to complicate this to "local user masks" and "local user unit files" and so on, and I am not sure the extra complication is really worth it.

I mean, every complication like this comes at a price. I am not sure the benefit in this case would clearly outweigh the price...

[poettering]

Actually I would say that adding Masked=true directive support would allow masking of any unit wherever it is located by appropriate drop-in.

When a unit file is masked we'll not read it at all, i.e. we won't parse it, and thus not create deps and stuff. With your approach we'd have to read it until we hit the "Masked=true" thing, but that would mean we'd always create After=/Before= and all the other deps, but then not honour them (or flush them again?), but that's just messy. I think masking things to /dev/null is nicely explanatory and makes clear the contents of the unit files is entirely ignored, and that's how masking should work, as the biggest gun in the belt.

[jnpkrn]

Another still slightly baroque (doesn't appear feasible with no footprint
at all) solution could be to handle self-blocking masking (I'd argue --user
units are impacted with this single-namespace-in-2-roles problem even more,
since unprivileged users cannot even use the single-namespace-single-role
designated directory/usr/lib/systemd/user) with maintaining a dedicated
tmpfiles.d config that would imitate the --runtime masking in /run/systemd/
along with triggering the effect of the actual systemctl mask --runtime.
This would presumably achieve the desired persistence of otherwise volatile
(runtime) configuration that would be abolished upon reboot otherwise.

Thankfully, one bug on the way there was fixed recently (#10583).

[keszybz]

I guess we could rename the original file, using a new extension, e.g. /etc/systemd/system/service.service.mask:

# systemctl mask foo
File /etc/systemd/system/foo.service already exists, saving as `/etc/systemd/system/foo.service.masked`.
Created symlink /etc/systemd/systemctl/foo.service → /dev/null.

and then

# systemctl unmask foo
Removed symlink /etc/systemd/systemctl/foo.service.
Renamed `/etc/systemd/system/foo.service.masked` to `/etc/systemd/system/foo.service`.

[vcaputo]

@keszybz That doesn't sound terrible to me. There are just some details surrounding the edge cases of the mask file already existing on systemctl mask, or missing on systemctl unmask which I don't see a perfect solution for. With the directory being the same one users are expected to fiddle around in, those cases seem more likely.

I think it's cleaner to always leave the services alone wherever they are, and just create the symlinks added on systemctl mask in isolation, but remove them on systemctl unmask wherever they exist.

@poettering If someone jumps into systemctl(1) for the express purpose of reading about the mask subcommand, this is what they find in v241:

       mask NAME...
           Mask one or more units, as specified on the command line. This will
           link these unit files to /dev/null, making it impossible to start
           them. This is a stronger version of disable, since it prohibits all
           kinds of activation of the unit, including enablement and manual
           activation. Use this option with care. This honors the --runtime
           option to only mask temporarily until the next reboot of the
           system. The --now option may be used to ensure that the units are
           also stopped. This command expects valid unit names only, it does
           not accept unit file paths.

Nothing there speaks to the inconsistent applicability to system-wide vs. local units.

I also find your threshold of baroqueness entirely arbitrary, making that argument rather empty.

If systemctl is intended to be used as the preferred interface for these operations, it should at least function consistently regardless of which of the myriad supported directories a unit happens to reside in.

[amiryal]

As a solution for sysadmins looking for a place to put their locally defined units, while allowing their users to modify them in local configuration, I suggest to add another level between /usr/systemd/system and /etc/systemd/system. Let’s call it /usr/local/lib/systemd/system?

[doubletwist13]

Well, masking a unit is done by symlinking it to /dev/null in /etc/systemd/, and that's documented. Which means you cannot really have both a file there and mask them, because you cannot have a single file be both a regular file and a symlink to /dev/null...

If we wanted to supported what you are asking for we'd have to add yet another directory we have in our search path, but I am not sure this is really helperful, as it makes things even more baroque to the user. Right now we distinguish only "local user configuration" stuff (i.e. in /etc) from "vendor" stuff (i.e. in /usr) basically. If we'd want to support the ability to mask user stuff, then we'd have to complicate this to "local user masks" and "local user unit files" and so on, and I am not sure the extra complication is really worth it.

I mean, every complication like this comes at a price. I am not sure the benefit in this case would clearly outweigh the price...

I'll put my vote in that it's worth the price of adding a separate directory if that's how you want masking to work.
As it stands now for custom service files we need to either:

1. Live without it
2. put the service file someplace OTHER than the 'officially' designated location for local service unit files. ie/usr/lib/systemd/system/ which appears to be only for package-installed service units.

As someone who strives for consistency and maintainability, it's frustrating to find that the recommended location for these files breaks functionality that we need.

[doubletwist13]

Idea: One option might be to add 1 dir to the search path (eg /etc/systemd/masks) designed to only contain the mask symlinks to >/dev/null that would overrides the other dirs (like /etc/systemd/system overrides /usr/lib/systemd/system).

That would leave the part that users/admins interact with regularly unchanged (location of custom unit files), and the masks, which I suspect most users/admins rarely if ever interact with directly, will have their own place that doesn't conflict with custom unit files.

[twbecker]

Minimally, there's a documentation issue here. Nothing in the man page even hints that this doesn't work for units in /etc/systemd/system, and when you try it, the error message is "baroque" to say the least. On systemd 237:

# systemctl mask foo.service
Failed to execute operation: Invalid argument

[mbiebl]

I guess we could rename the original file, using a new extension, e.g. /etc/systemd/system/service.service.mask:

I understand @poettering 's concern about not wanting to parse masked services at all.

Another idea, which would complement the .service.conf.d idea quite nicely would be, not to rename the file and then creating a symlink, but simply creating a file foo.service.mask (and removing that (empty) file on unmask). The existence of such a file would be an indicator that the file is masked.

[alexp1917]

It would be great i this was addressed, using any of the proposals listed. I have just moved a file into the etc/systemd folder just to be able to run mask on the service to disable it.