zfs datasets do "not stay mounted" / seem to be mounted for current user session only

[deajan]

Hello,

This might be a bit odd, never experienced this problem with ZFS before, but here we go.

I've created a fresh zpool via

zpool create -o ashift=12 ice raidz1 scsi-[...] scsi-[...] scsi-[...] scsi-[...]y

Once this is done, I can see the new mountpoint via

mount | grep zfs
ice on /ice type zfs (rw,seclabel,xattr,noacl)

Everything works well so far, but I cannot use that mountpoint in any systemd service, nor cron job since those jobs won't find the FS, and use the empty mountpoint /ice which will not be zfs.

Also, as soon as I logout from my user and login again (ssh), the FS is not mounted anymore, and I have to run zfs mount ice in order to get the mountpoint again.

I've been running ZoL since the 0.6x days, and never ever got any of these issues before.

Any guidance would be appreciated.

Diag info:
RHEL 9.2 linux 5.14.0-284.25.1.el9_2.x86_64
ZoL 2.1.12-1 (dkms)

systemctl | grep zfs
  zfs-import-cache.service                                                                         loaded active exited    Import ZFS pools by cache file
  zfs-mount.service                                                                                loaded active exited    Mount ZFS filesystems
  zfs-share.service                                                                                loaded active exited    ZFS file system shares
  zfs-zed.service                                                                                  loaded active running   ZFS Event Daemon (zed)
  zfs-import.target                                                                                loaded active active    ZFS pool import target
  zfs.target                                                                                       loaded active active    ZFS startup target

Nothing relevant in log files.

I noticed something strange though, if I happen to run systemctl restart zfs-mount, then I get my zfs mounted twice:

mount | grep zfs
ice on /ice type zfs (rw,seclabel,xattr,noacl)
ice on /ice type zfs (rw,seclabel,xattr,noacl)

Is there some "mount as current user" magic involved I am not aware of ?

[rincebrain]

They're probably not mounted in whatever namespace(s) systemd is running the services in.

[deajan]

Makes sense.
But out of the systemd namespaces, when I run zfs mount my/dataset then logout and login, the mountpoint doesn't stay.
That can't be explained by namespaces don't it ?

[deajan]

I've tested two systems with identical platform (zfs 2.1.12 on RHEL 9.2).
On one of the systems, zfs behaves like it should, on the other, mountpoints dissapear after logout unless I use systemctl restart zfs-mount.

The latter system is hardened with ANSSI BP-028 High profile, but I can't find any parameter that could influence this behavior.
Any directions to search for perhaps ?

[rincebrain]

Set Up a Private Namespace in PAM Configuration

I would guess that's giving you a new namespace without the relevant mounts in it for each login.

I don't know how to correctly set it up, as it's not a knob I've ever seen used before, but if I had to speculate, that'd be where I'd look.

[deajan]

Thanks, but both systems are identically configured to use pam namespaces (actually a RHEL 9 default).
So /etc/pam.d/login contains session required pam_namespace.so on both systems.
I've checked with a third (vanilla) system, which has the same setting.

but you're right on the mount namespaces.
I've created a new zfs dataset on both systems and checked the mountpoint stats using grep <fsname> /proc/self/mountinfo

System on which zfs mountpoints are available to every user:

1431 98 0:84 / /stor/toto rw shared:611 - zfs stor/toto rw,seclabel,xattr,noacl

System on which zfs mountponits are available to current session only:

1546 1396 0:73 / /ice/toto rw,noatime - zfs ice/toto rw,seclabel,xattr,noacl

If I happen to run systemctl restart zfs-mount, only the later system mounts change:

1546 1592 0:73 / /ice/toto rw,noatime - zfs ice/toto rw,seclabel,xattr,noacl
1592 1396 0:73 / /ice/toto rw,noatime master:342 - zfs ice/toto rw,seclabel,xattr,noacl

I guess I don't inherit the MS_SHARED default attribute for mount points.

I could stop here and say horray, but I'm curious.
I've grepped all sysctl values and /etc folder searching for 'mount' and 'shared' text, without luck.
I've also looked in the ANSSI BP-028 document, searching for the same texts, without luck too.
I've even googled about mount namespace defaults, without much luck.

Any idea ?
I thought I wasn't that bad at being a linux admin, but here I am,

[rincebrain]

pam_namespace is enabled by default, but getsebool polyinstantiation_enabled is off by default, unlike in that setup.

[deajan]

While I would have liked this to be the reason, polyinstantiation is only used for /var and /var/tmp AFAIK, and would not affect generic mounts.