Centos: systemd-journald.service misses the zfs-mount.service dependency

[drook]

System information

Type	Version/Name
Distribution Name	Centos
Distribution Version	7.5
Linux Kernel	3.10.0-862.14.4
Architecture	amd64
ZFS Version	0.7.11-1
SPL Version	0.7.11-1

Describe the problem you're observing

systemd-journald.service misses the zfs-mount.service dependency. Thus, when /var/log is on a separate dataset (pretty much the standard nowadays) at the moment zfs-mount service is starting, /var/log isn't empty, because journald has already put the files there.

Describe how to reproduce the problem

Make /var/log reside on a separate dataset inside the root pool, and attempt to boot the system. Some of jourlald services (socket, flush-to-persistent-storage and journald itself will be in a failed state).

Include any warning/errors/backtraces from the system logs

[mskarbek]

This is not an issue for ZoL just a matter of proper configuration on your side. Simply put override file for each unit that depends on your mount point:

[Unit]
Requires=zfs-mount.service
After=zfs-mount.service

For journald that will be: /etc/systemd/system/systemd-journald.service.d/override.conf

[drook]

Well, you're right - yeah, that's a minor issue, but wouldn't it be great if this override.conf file was included in the centos package ?

[mskarbek]

Nope, the fact that you have /var/log configured as a ZFS dataset doesn't mean that everyone has it that way. Shipping such override.conf with the package would cause more damage than good.

[drook]

Same rule applies to your approach - the fact that you don't have the /var/log on a separate dataset doesn't mean everyone shouldn't have it there. In fact, most modern installers provided with ZFS-enabled OSes do configure the /var/log on a separate dataset: Solaris does, FreeBSD does.... welll, that's the end of list for now, but this pretty much means that any user migrating his habits from these OSes to Linux will want /var/log on a separate dataset (I could go much further and say that I believe everyone should have the /var/log on a separate dataset, but that would be counterproductive).

And what harm could be done by starting journald after the zfs-mount.service ? I see none. I see that the defaults provided currently are non-universal and hard to fix by newbies.

[mskarbek]

Do you ship predefined /etc/fstab? No, that's something local, generated during installation and modified after that accordingly to the changing situation. The same goes for messing with the configuration of others services, you are not packaging that, it's a matter of local requirements, specific to that concrete installation and to be provided by administrator or automation script after installation. Linux is not a native environment for ZFS, nobody expects and tests changes from ZFS besides ZoL users so comparing to the ZFS-enabled OSes is not adequate. If you are expecting such configuration to be provided automatically on a Linux installation with ZFS on root for example then it's a matter of instillator to provide that change.

[rlaager]

Rather than adding an override for each unit, you can make /var/log a legacy mount and put it in /etc/fstab. In this way, things lean on the built-in/default mount dependencies. Once ZFS has a systemd mount generator, it will not be necessary to use legacy mounts.

I have this in the Ubuntu 18.04 HOWTO:
https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS

# zfs set mountpoint=legacy rpool/var/log
# zfs set mountpoint=legacy rpool/var/tmp
# cat >> /etc/fstab << EOF
rpool/var/log /var/log zfs noatime,nodev,noexec,nosuid 0 0
rpool/var/tmp /var/tmp zfs noatime,nodev,nosuid 0 0
EOF

If you created a /tmp dataset, do the same for it:
# zfs set mountpoint=legacy rpool/tmp
# cat >> /etc/fstab << EOF
rpool/tmp /tmp zfs noatime,nodev,nosuid 0 0
EOF

[rlaager]

ZFS has a systemd mount generator these days.