systemd fails to parse mount unit drop-in configs when the unit was detected as not-found previously

[stewartadam]

Submission type

• Bug report

systemd version the issue has been seen with

systemd-237 (per this comment) and systemd-234-10.git5f8984e.fc27.x86_64

This bug report is a port of the issue originally reported on the Red Hat Bugzilla as #1560234 as it has been confirmed to still occur with upstream system.

Under very specific conditions, I've noticed that systemd will fail to parse drop-in configs at all - it looks like an ordering issue with the build-in mount unit generator. Those conditions are:

• A mountpoint has drop-in config files
• Service 'X' is enabled
• Service 'X' lists ANY dependency (Before, After or Requires) on said mountpoint

Used distribution

Fedora 27

In case of bug report: Expected behaviour you didn't see

Drop-in configs are parsed, and foo.service is started. This can be realized by omitting step 2.

$ systemctl status mnt-tmp.mount
● mnt-tmp.mount - /mnt/tmp
   Loaded: loaded (/proc/self/mountinfo)
  Drop-In: /etc/systemd/system/mnt-tmp.mount.d
           └─start-foo.conf
   Active: active (mounted) since Sat 2018-03-24 22:32:40 PDT; 65ms ago
    Where: /mnt/tmp
     What: mnt/tmp
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/mnt-tmp.mount

In case of bug report: Unexpected behaviour you saw

Drop-in configs are not parsed missing and systemd does not report cgroup.

$ systemctl status mnt-tmp.mount
Warning: mnt-tmp.mount changed on disk. Run 'systemctl daemon-reload' to reload units.
● mnt-tmp.mount - /mnt/tmp
   Loaded: loaded
   Active: active (mounted) since Sat 2018-03-24 22:32:25 PDT; 17ms ago
    Where: /mnt/tmp
     What: mnt/tmp

Note how this mount unit (1) has not parsed the drop-ins, requesting daemon reload (2) is not based on /proc/self/mountinfo (3) has no cgroup.

In case of bug report: Steps to reproduce the problem

1. Setup a foo.service with a dependency on mnt-tmp.mount:

cat << EOF > /etc/systemd/system/foo.service
# Stub service for testing
[Service]
Type=simple
ExecStart=/bin/true

Requires=mnt-tmp.mount
After=mnt-tmp.mount

[Install]
WantedBy=multi-user.target
EOF

2. Enable the foo service:

systemctl enable foo.service

3. Configure mnt-tmp.mount to pull foo.service service upon mount:

mkdir /etc/systemd/system/mnt-tmp.mount.d
cat << EOF /etc/systemd/system/mnt-tmp.mount.d/start-foo.conf
# Pull in 'foo' service after mounting /mnt/tmp
[Unit]
Before=foo
Wants=foo
EOF
systemctl daemon-reload

4. Trigger a mount of /mnt/tmp:

mount -o size=1M -t tmpfs tmpfs /mnt/tmp

Additional info:

Context for this issue is that I have encrypted filesystems that are loaded manually post-boot. Some service depend on these mounts, so I want the services to be started automatically once they do eventually get mounted (via a Wants= and Before= in encrypted-fs.mount.d drop-in config).

This works but the service still happily starts without the mounted fs -- so I have also add a Requires=encrypted-fs.mount and After=encrypted-fs.mount in the service drop-in config. It is at this point that you will experience failure if the service is enabled. It appears to work fine if disabled.

[keszybz]

This issue is caused by a misunderstanding:

• Mount units require a .mount file in general (because systemd needs to know what to mount where, "where" is given by the mount unit name, but not "what").
• When an attempt is made to load a unit, and this unit requires a file, but the file is not found (even though drop-ins may exist), operation is failed as "not found".
• There is an exception to the first rule: mount units created based on existing mount points in /proc/self/mountinfo do not require a file.
• Enabling foo.service which requires mnt-tmp.mount means that the both those units will be loaded early in the transaction, and loading of mnt-tmp.mount will fail because the main unit file does not exist.

Thus, if you want to use /etc/systemd/system/mnt-tmp.mount.d/start-foo.conf, then mnt-tmp.mount file must exist somewhere. AFAICT, there is not bug.

There's a minor buglet: it seems pointless to load drop-ins for mount points created from /proc/self/mountinfo, since by the time they are read, the mount point is already active. But it seems like a very edge case, so I wouldn't spend energy on fixing this unless somebody shows that this matters somehow.

[stewartadam]

Hi Zbigniew ,

Thus, if you want to use /etc/systemd/system/mnt-tmp.mount.d/start-foo.conf, then mnt-tmp.mount file must exist somewhere. AFAICT, there is not bug.

I don't disagree here -- the Requires=mnt-tmp.mount and After=mnt-tmp.mount are there as a failsafe to prevent the service from activating without those mountpoints present (on boot if enabled, I would expect it to not find any mount files and therefore fail as you describe; although that doesn't happen due to linked bug above).

it seems pointless to load drop-ins for mount points created from /proc/self/mountinfo

Here is where I respectfully disagree, as I have the intended behavior working with drop-ins for mount points created from /proc/self/mountinfo: when those mount points are created on-the-fly, the drop-ins are read which pulls in eg. foo.service using a Before= and Wants=; net result is that when I do manually mount those filesystems, the services are started.

Regardless, I would expect that even if this is decided as something to be unsupported that systemd's dependency solving or config loading behavior for mount units would not vary when depending on if a depending service is enabled/disabled.

[keszybz]

It does not "vary on whether it is enabled/disabled", it varies based upon when and how the service is loaded. When it is loaded determines how it is loaded. When it is loaded before mounting, it fails to load, like described above. When it is loaded after mounting, i.e. synthetized from /proc/self/mountinfo, it "loads" successfully.

We could make this "consistent" by disallowing mounts to be created from /proc/self/mountinfo, i.e. by forbidding mount calls that do not go through systemd. I don't think anybody would like that. OTOH, normal mount units require What= to be specified, so it's reasonable to require the main file to be present. (We could allow just drop-ins without the main file, assuming that What= would be specified somewhere, but that would be even more confusing and wouldn't really change anything.)

Here is where I respectfully disagree, as I have the intended behavior working with drop-ins for mount points created from /proc/self/mountinfo: when those mount points are created on-the-fly, the drop-ins are read which pulls in eg. foo.service using a Before= and Wants=; net result is that when I do manually mount those filesystems, the services are started.

You're using some corner-cases of the implementation. In particular Before= cannot be realized — after all, once the mount is detected, it is already too late. Hence, I think it'd be more robust and predictable to use normal .mount units for this, and allow systemd to resolve the dependencies.

[filbranden]

@stewartadam You might also want to look at systemd-mount.

[arvidjaar]

@keszybz

Enabling foo.service which requires mnt-tmp.mount means that the both those units will be loaded early in the transaction,

I am not sure I understand what transaction you mean here. There is no reboot; there is only systemctl daemon-reload which does not initiate any transaction to my best knowledge.

[stewartadam]

@filbranden thanks for the suggestion -- I have looked at that, although it would work it would also require I create (and maintain) copies of mount units for every filesystem I create. Auto-generated mount units with drop-in configs maintains the ideal behavior, allowing for starting start of services when the mount happens, whenever that may be.

Because these are non-root encrypted filesystems for which I want to manually enter the key (but not block boot on), the scheduling aspect of systemd-mount also provides me with little benefit compared to autogenerated mounts.

It does not "vary on whether it is enabled/disabled", it varies based upon when and how the service is loaded. [...] We could make this "consistent" by disallowing mounts to be created from /proc/self/mountinfo

@keszybz I suppose what I am saying is that as a user, it seems bizarre that a failure to load a non-existent unit would affect the behavior of a unit by the same name should it be actually generated in the future, as we see here for these /proc/self/mountinfo-based mounts.

Instead of disabling all /proc/self/mountinfo mounts, my suggestion would be to reload the unit if we detect a unit previously not found was generated (or otherwise treat the generated unit and not found unit separately). That said, I think I've made my position sufficiently clear so I'll respect your decision if you still think this is a wontfix.

[keszybz]

I am not sure I understand what transaction you mean here.

Yeah, "transaction" wasn't the right word. I was thinking of behaviour that would be observed after a reboot, which would exhibit the same effect as the reproducer shown above.

I suppose what I am saying is that as a user, it seems bizarre that a failure to load a non-existent unit would affect the behavior of a unit by the same name should it be actually generated in the future, as we see here for these /proc/self/mountinfo-based mounts.

OK, that is a good point. We should load drop-ins when generating a mount based on /proc/self/mountinfo, even if the unit was not-found previously.