man: document explicitly that unit files on mount points that are established at late boot cannot be read

[Lahorde]

Submission type

• Bug report

systemd version the issue has been seen with

systemctl --version
systemd 236
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

Used distribution

Arch

Expected behaviour you didn't see

After enabling a unit I can start it. But after a reboot, I cannot anymore start it. Unit file is not found. It is still present in /etc/sytemd/system
I can disable it.
In my case, unit located in some non root folder. It has following permissions :

ls -l ./pws_client.service 
-rw-r--r-- 1 root root 563 28.02.2018 10:49 ./pws_client.service

A solution is :

• to copy unit file to /etc/systemd/system unit is started after reboot.
• copy /etc/systemd/system to /etc and enable it

Unit is located in /home partition that is different from / partition. A possible explanation could be that when starting, systemd does not see my unit on unmounted /home partition. It would explain why after startup when systemctl daemon-reload unit becomes "visible" by systemd and I can start it.
Is it a valid explanation?
Something strange is that I always have / in a different partition than /home, and I often symlink unit files located in /home. And as you can see from logs, for unit monitor_hot_plug.service I do not have any issue: service is found, I can start it manually after a reboot. The difference between services is that monitor_hot_plug.service is not enabled.
Does that mean when a unit is started and not found, it won't be visible by systemd until next daemon-reload ?

Steps to reproduce the problem

I can enable and start service :

sudo systemctl enable /home/remi/projects/perso/pws_widget/pws_client.service
Created symlink /etc/systemd/system/graphical.target.wants/pws_client.service → /home/remi/projects/perso/pws_widget/pws_client.service.
Created symlink /etc/systemd/system/pws_client.service → /home/remi/projects/perso/pws_widget/pws_client.service.
remi@remi-dell-5520 ~ $ sudo systemctl start pws_client && systemctl status pws_client
● pws_client.service - PWS client. Gathered weather, pollution, and others data from WU, ATMO, PiouPiou
   Loaded: loaded (/home/remi/projects/perso/pws_widget/pws_client.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-02-28 10:50:06 CET; 6ms ago
 Main PID: 30933 (start_pws_polli)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/pws_client.service
           └─30933 /bin/bash /home/remi/projects/perso/pws_widget/start_pws_polling.sh

févr. 28 10:50:06 remi-dell-5520 systemd[1]: Started PWS client. Gathered weather, pollution, and others data from WU, ATMO, PiouPiou.
févr. 28 10:50:06 remi-dell-5520 sh[30933]: reading pws client parameters
févr. 28 10:50:06 remi-dell-5520 sh[30933]: 2018-02-28 10:50 06 - start_pws_polling.sh - start pws data fetch

...

After a reboot, unit is not found anymore.

systemctl status pws_client
Unit pws_client.service could not be found.
remi@remi-dell-5520 ~ :( $ systemctl status pws_client
Unit pws_client.service could not be found.
remi@remi-dell-5520 ~ :( $ sudo systemctl start pws_client
[sudo] Mot de passe de remi : 
Failed to start pws_client.service: Unit pws_client.service not found.                    
remi@remi-dell-5520 ~ :( $ ls -l /etc/systemd/system/
total 32
drwxr-xr-x 2 root root 4096 04.12.2017 22:13 bluetooth.target.wants/
drwxr-xr-x 2 root root 4096 04.12.2017 21:41 getty.target.wants/
drwxr-xr-x 2 root root 4096 28.02.2018 10:49 graphical.target.wants/
drwxr-xr-x 2 root root 4096 15.12.2017 12:24 multi-user.target.wants/
drwxr-xr-x 2 root root 4096 15.12.2017 12:21 printer.target.wants/
drwxr-xr-x 2 root root 4096 15.12.2017 12:24 sockets.target.wants/
drwxr-xr-x 2 root root 4096 04.12.2017 22:43 sysinit.target.wants/
lrwxrwxrwx 1 root root   41 04.12.2017 22:13 dbus-org.bluez.service -> /usr/lib/systemd/system/bluetooth.service
lrwxrwxrwx 1 root root   44 15.12.2017 12:24 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx 1 root root   46 04.12.2017 22:49 dbus-org.freedesktop.NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx 1 root root   57 04.12.2017 22:49 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx 1 root root   48 05.12.2017 23:36 display-manager.service -> /usr/lib/systemd/system/lightdm-plymouth.service
lrwxrwxrwx 1 root root   73 23.01.2018 10:38 monitor_hot_plug.service -> /home/remi/projects/github_gists/monitor_hotplug/monitor_hot_plug.service
lrwxrwxrwx 1 root root   55 28.02.2018 10:49 pws_client.service -> /home/remi/projects/perso/pws_widget/pws_client.service
remi@remi-dell-5520 ~ $ systemctl status monitor_hot_plug.service
● monitor_hot_plug.service - Monitor hotplug
   Loaded: loaded (/home/remi/projects/github_gists/monitor_hotplug/monitor_hot_plug.service; linked; vendor preset: disabled)
   Active: inactive (dead)

févr. 28 10:51:19 remi-dell-5520 monitor_hotplug.sh[528]: Invalid MIT-MAGIC-COOKIE-1 keyCan't open display :0
févr. 28 10:51:19 remi-dell-5520 monitor_hotplug.sh[528]: Number of connected screen at sys level : 2 - at xrandr level : 0
févr. 28 10:51:20 remi-dell-5520 monitor_hotplug.sh[528]: number of connected screens = 2
févr. 28 10:51:20 remi-dell-5520 monitor_hotplug.sh[528]: connect monitor DP-1-1
févr. 28 10:51:21 remi-dell-5520 monitor_hotplug.sh[528]: conky: aucun processus trouvé
févr. 28 10:51:32 remi-dell-5520 systemd[1]: Started Monitor hotplug.
févr. 28 10:51:34 remi-dell-5520 monitor_hotplug.sh[3599]: number of connected screens = 1
févr. 28 10:51:38 remi-dell-5520 systemd[1]: Started Monitor hotplug.
févr. 28 10:51:43 remi-dell-5520 monitor_hotplug.sh[4248]: number of connected screens = 2
févr. 28 10:51:43 remi-dell-5520 monitor_hotplug.sh[4248]: connect monitor DP-1-1
remi@remi-dell-5520 ~ :( $ sudo systemctl start monitor_hot_plug.service && systemctl status monitor_hot_plug.service
● monitor_hot_plug.service - Monitor hotplug
   Loaded: loaded (/home/remi/projects/github_gists/monitor_hotplug/monitor_hot_plug.service; linked; vendor preset: disabled)
   Active: active (running) since Wed 2018-02-28 10:52:59 CET; 5ms ago

But I can disable it. A solution is to copy it.

remi@remi-dell-5520 ~ $ sudo systemctl disable pws_client
Removed /etc/systemd/system/pws_client.service.
Removed /etc/systemd/system/graphical.target.wants/pws_client.service.
remi@remi-dell-5520 ~ $ sudo cp /home/remi/projects/perso/pws_widget/pws_client.service /etc/systemd/system
remi@remi-dell-5520 ~ $ sudo systemctl enable pws_client
Created symlink /etc/systemd/system/graphical.target.wants/pws_client.service → /etc/systemd/system/pws_client.service.

[poettering]

Sorry, but unit files have to reside on a partition that is mounted at the moment the host PID 1 is invoked. i.e. either on the root partition or some other partition that the initrd premounts.

PID 1 will determine the full start transaction at early boot and than start it as parallellized as possible on one go. If the unit files cannot be read then they cnanot be included in the startup transaction.

It's fine if the invoked programs themselves are only available during late boots, but the unit files need to be there from the earliest instance on.

[poettering]

I figure we should document this behaviour better though

[sourcejedi]

+5 from https://unix.stackexchange.com/questions/291085/systemd-linked-unit-files-on-mounted-disk-fail-to-load (and one more effective duplicate question).

[gunar]

So this is a WONTFIX? It's really annoying. Is there a workaround? I want to keep my unit files alongside my dotfiles in a git repo in /home.

[sourcejedi]

You need to have your config files mounted, at the time you run the daemon that they apply to.

Also, systemd does not currently have any feature, which will start newly added units if they happen to depend on the target that was started initially. So you can't just wedge a daemon-reload in part-way through boot, and have it fix things up as if they were mounted all along.

Apart from that limitation, daemon-reload partway through boot might be expected to work well enough since e266c06. That suggests some possibilities for a workaround (if you're not talking about encrypted homedirs!). But IMO they are too ugly; I do not detail them further.

If you really can't tolerate either using an install script, or creating a dedicated directory in the / filesystem, a workaround for a straightforward /home filesystem would be if you use an initrd which supports the x-initrd.mount option for such a filesystem. (Such initrd's read /etc/fstab after mounting the target / filesystem). That will allow you to get all your config files for systemd mounted before the initrd starts systemd from the / (or /usr) filesystem.

I think those are the main options you need to consider.

[j0057]

Here's some further detail as to what works for me to get timers running from a dotfiles repo in my homedir which sits on a separate partition from the root partition:

[Unit]
After = local-fs.target

[Service]
Type = oneshot
ExecStart = systemctl daemon-reload
ExecStart = systemctl start timers.target

[Install]
WantedBy = multi-user.target```