remote-fs.target not active when upgrading from older systemd versions

[a2yp]

Description

Since version 2605.5.0 NFS shares are not mounted after reboots.

Impact

NFS share is not mounted after reboot and upgrade.

Environment

flatcar-install -d /dev/sda -i ignition.json -o vmware_raw
NFS share: https://docs.flatcar-linux.org/os/mounting-storage/#mounting-nfs-exports

steps to reproduce

cat /etc/systemd/system/mnt-nfs.mount

[Unit]
Before=remote-fs.target
[Mount]

What=<ip_of_nfs_server>:/nfs

Where=/mnt/nfs

Type=nfs
[Install]

WantedBy=remote-fs.target

ls -l /etc/systemd/system/remote-fs.target.wants/

total 4
lrwxrwxrwx. 1 root root 33 Sep 22 11:29 mnt-nfs.mount -> /etc/systemd/system/mnt-nfs.mount

systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: active (mounted) since Tue 2020-09-22 11:50:06 UTC; 31min ago
      Where: /mnt/nfs
       What: :/nfs
      Tasks: 0 (limit: 28807)
     Memory: 356.0K
     CGroup: /system.slice/mnt-nfs.mount

reboot

systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: inactive (dead)
      Where: /mnt/nfs
       What: :/nfs

sudo systemctl start mnt-nfs.mount

systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: active (mounted) since Tue 2020-09-22 12:38:00 UTC; 6s ago
      Where: /mnt/nfs
       What: :/nfs
      Tasks: 0 (limit: 28807)
     Memory: 344.0K
     CGroup: /system.slice/mnt-nfs.mount

Expected behavior

NFS share is mounted after reboot and upgrade.

[margamanterola]

The upgrade to 2605.5.0 meant upgrading to systemd 245. It's likely that there's some difference there causing your unit not to start automatically. Could you check the status of remote-fs.target?

[a2yp]

@marga-kinvolk

systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2020-09-22 12:50:50 CEST; 2h 18min ago
       Docs: man:systemd.special(7)

[margamanterola]

Right, it's disabled. It's a consequence of this change:

        * During package installation (with `ninja install`), we would create
          symlinks for getty@tty1.service, systemd-networkd.service,
          systemd-networkd.socket, systemd-resolved.service,
          remote-cryptsetup.target, remote-fs.target,
          systemd-networkd-wait-online.service, and systemd-timesyncd.service
          in /etc, as if `systemctl enable` was called for those units, to make
          the system usable immediately after installation. Now this is not
          done anymore, and instead calling `systemctl preset-all` is
          recommended after the first installation of systemd.

(From https://github.com/systemd/systemd/blob/master/NEWS)

We solved this for getty in flatcar-archive/coreos-overlay#400, I guess we need to solve it for remote-fs.target as well.

In the meantime, you can enable the unit yourself to solve the issue for your instances.

[t-lo]

On a fresh (qemu) Stable-2605 installation I see:

localhost ~ # systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: disabled)
     Active: active since Tue 2020-09-22 13:06:28 UTC; 8min ago
       Docs: man:systemd.special(7)

@a2yp just to narrow down the symptoms, did you upgrade or deploy a fresh instance?

[a2yp]

@t-lo The instance automatically upgraded from 2512.5.0 to 2605.5.0.

[margamanterola]

I installed a qemu instance with 2512.5.0 and upgraded and I could reproduce this issue. With 2512:

core@localhost ~ $ systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
   Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
   Active: active since Tue 2020-09-22 13:54:52 UTC; 32s ago
     Docs: man:systemd.special(7)

Notice how it says it's disabled, but still active. And there are no events (it's not that I forgot to copy them, there were no events).

After upgrading and rebooting:

core@localhost ~ $ systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2020-09-22 14:03:22 UTC; 2min 28s ago
       Docs: man:systemd.special(7)

Sep 22 14:03:22 localhost systemd[1]: Reached target Remote File Systems.
Sep 22 14:03:22 localhost systemd[1]: Stopped target Remote File Systems.

Now it's disabled and inactive, but we do see a couple of events. So, maybe something else is at play.

[margamanterola]

This is relevant, I think.

In the previous stable:

Flatcar Container Linux by Kinvolk oldstable (2512.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target
/usr/lib/systemd/system/multi-user.target.wants/remote-fs.target
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount

In the current stable (after upgrading):

Flatcar Container Linux by Kinvolk stable (2605.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target

So, /usr/lib/systemd/system/multi-user.target.wants/remote-fs.target is missing.

[margamanterola]

And, for completion, on a freshly installed machine:

Flatcar Container Linux by Kinvolk stable (2605.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target

This is of course the same, BUT we have this file which is not present in the upgrade machine:

core@localhost ~ $ find /etc/systemd/ | grep remote-fs
/etc/systemd/system/multi-user.target.wants/remote-fs.target

[margamanterola]

I'm not sure why multi-user.target.wants/remote-fs.target ended up in /etc/ instead of /usr, but I think that's what we should fix. It should be in /usr so that it reaches all users, not just the fresh installs.

(Same goes for getty.target.wants/getty@tty1.service)

[KillahB33]

Thanks for this got my mounts fixed now! I'll update as soon as the permanent fix get's pushed.