Snapshot automount (.zfs) still fails under chroot (OpenSSH ChrootDirectory)

[mattgarber]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	9.9 (Stretch)
Linux Kernel	4.9.0-9-amd64
Architecture	amd64
ZFS Version	0.8.0-1
SPL Version	0.8.0-1

Describe the problem you're observing

When attempting to browse auto-mounting ZFS dataset snapshots (.zfs/snapshot directory) over a chrooted SFTP (OpenSSH’s ChrootDirectory directive), snapshot access fails with the following error:

WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512

Browsing snapshots in this manner works as intended if ChrootDirectory is not set (not a workable solution, due to needing to restrict SFTP client access to the filesystem), and also works correctly on FreeBSD (11.2-RELEASE, 12.0-RELEASE), presumably because of differing methods of auto-mounting snapshot access.

This appears to be extremely similar to #3778, and @behlendorf indicated various fixes landing in 0.6.x and 0.7.x, although perhaps it wasn’t fully resolved for all kinds of chroot? The error is clear, in that the mount operation is being called on the SFTP client’s view of the filesystem (where /tank/backups/test01 appears as /), even though it should be resolved outside of the chroot.

I first ran into this issue on Ubuntu 18.04, running distro-provided zfs/spl 0.7.5, although attempted this reproduction on Debian with the latest 0.8.0 release to see whether the #3778 discussion/fixes were landed after Ubuntu’s version.

Describe how to reproduce the problem

1. Create a new dataset for hosting restricted client content, e.g. tank/backups/test01, mounted at /tank/backups/test01.

2. Create some dummy files, and create a symlink at the top of the dataset, pointing to the hidden snapshots directory: ln -s .zfs/snapshot snapshots. (The symlink is only so that SFTP clients don’t have to explicitly view hidden files, it’s not a prerequisite for reproducing the issue!)

3. Create a few snapshots of the dataset: zfs snapshot tank/backups/test01@one, zfs snapshot tank/backups/test01@two.

4. Create a test user (e.g. test01) and configure OpenSSH to restrict access for that particular user, including ChrootDirectory, and restart sshd. E.g.:

Match User test01
	ChrootDirectory /tank/backups/test01
	ForceCommand internal-sftp -l INFO -u 022
	PasswordAuthentication yes

5. Connect via SFTP as specified test user. Verify that dummy files and directories are successfully browsable.

6. Change into the snapshots symlink (.zfs/snapshot), and verify that snapshots list is accurate. (The symlink is only so that SFTP clients don’t have to explicitly view hidden files, it’s not a prerequisite for reproducing the issue!)

7. Attempt to enter one or more snapshot directories. No content will appear, and kernel log on host system will include one or more errors similar to above: WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512.

Include any warning/errors/backtraces from the system logs

[ 5061.371785] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5061.457781] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5061.464070] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.660823] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.747940] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.757878] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512

[johnnyjacq16]

Run zfs get snapdir,snapdev <your dataset> the value for snapdir should be visible, the default is hidden.

[mattgarber]

# zfs get snapdir,snapdev tank/backups/test01

NAME                 PROPERTY  VALUE    SOURCE
tank/backups/test01  snapdir   visible  local
tank/backups/test01  snapdev   hidden   default

In /var/log/messages:

[70421.106185] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512
[70421.201172] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512
[70421.208336] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512

[johnnyjacq16]

for logging run journalctl -k -b 0 -f
then in another terminal try to mount the snapshot manually with mount -t zfs <the dataset snapshot> <a directory>

[mattgarber]

Mounting all snapshots manually (mount -t zfs <snapshot> <mountpoint>), or auto-mounted via the .zfs/snapshot directory, works completely fine on the host itself (not chrooted), or over SFTP when ChrootDirectory isn’t being used.

This issue is only occurring when trying to auto-mount the snapshots through a chrooted SFTP, because the auto-mounter is using the client’s view of the filesystem (chrooted to /) instead of the true filesystem location. And obviously, the .zfs/snapshot directory isn’t really at /.zfs, it’s at /tank/backups/test01/.zfs.

[mattgarber]

@behlendorf In #3778, you mentioned the ability to get full mount path in a chroot was resolved “in the 0.7 series,” but I didn’t see any specific linked commits, or other indication of when exactly it landed in 0.7 (comment date puts it at 0.7.6 or prior). Was it reverted out at some point, or does this potentially appear to be a different enough circumstance based on how OpenSSH might be performing the user chroot?

I also noticed somewhat similar issue #3512 regarding Linux namespace usage preventing full path resolution of mounts or snapshots in containers, although in this case OpenSSH should only be chrooting the user, not in a different namespace, so I’m a little stumped – not running in any containers here.

[behlendorf]

@mattgarber multiple issues were resolved regarding the automounting of snapshots. Unfortunately, automounting in your circumstance appears not to be handled. As you correctly pointed it, it appears to be because the zfsctl_snapshot_path() function is generating the wrong full path for the mount when triggered in the chroot environment. One possible way to handle this could be to fallback to the full path returned by zfsctl_snapshot_path_objset() when the zfsctl_snapshot_path() clearly isn't valid.

[mattgarber]

@behlendorf Thanks for taking a look and confirming this as a different case. Do you remember if there was a reason the zfsctl_snapshot_path_objset() fallback wasn’t added during the #3778 discussion, such as a performance hit? Or just dropped due to the other fixes at the time for automounting in separate cases (e.g. NFS)?

Is there an easy/good way to “clearly” determine the path isn’t valid, or is the best solution to do a quick compare between the values of zfsctl_snapshot_path() and zfsctl_snapshot_path_objset(), and using the value of zfsctl_snapshot_path_objset() as the mountpoint if there’s a mismatch? (I’m not sure if that would then cause any other kind of regression…)

[behlendorf]

@mattgarber if possible, would you mind verifying the proposed fix in #8966.

[mattgarber]

@behlendorf @kusumi Thank you! Was just able to confirm on a fresh installation that the proposed fix #8966 eliminates the issue preventing automounting snapshots in this chroot.

sftp> ls -lh /
lrwxrwxrwx    ? 0        0             13B Jun 29 22:31 /snapshots
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test1
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test2
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test3
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test4
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test5
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test6
sftp> ls -lh /snapshots/
drwxrwxrwx    ? 0        0              0B Jun 29 22:31 /snapshots/one
drwxrwxrwx    ? 0        0              0B Jun 29 22:39 /snapshots/two
sftp> ls -lh /snapshots/one/
lrwxrwxrwx    ? 0        0             13B Jun 29 22:31 /snapshots/one/snapshots
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test1
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test2
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test3
sftp> 

tank/backups/test01@one   97G  128K   97G   1% /tank/backups/test01/.zfs/snapshot/one