Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automount fails when accessing snapshots for root filesystem #9381

Closed
janlam7 opened this issue Sep 30, 2019 · 9 comments
Closed

Automount fails when accessing snapshots for root filesystem #9381

janlam7 opened this issue Sep 30, 2019 · 9 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@janlam7
Copy link

janlam7 commented Sep 30, 2019

System information

Type Version/Name
Distribution Name Gentoo
Linux Kernel 4.19.72-gentoo
Architecture x86_64
ZFS Version 0.8.2
SPL Version 0.8.2
# zfs get all bto/system/rootfs 
NAME               PROPERTY                      VALUE                         SOURCE
bto/system/rootfs  type                          filesystem                    -
bto/system/rootfs  creation                      Fri Jan 12  9:44 2018         -
bto/system/rootfs  used                          17.1G                         -
bto/system/rootfs  available                     5.33G                         -
bto/system/rootfs  referenced                    4.68G                         -
bto/system/rootfs  compressratio                 2.04x                         -
bto/system/rootfs  mounted                       yes                           -
bto/system/rootfs  quota                         none                          default
bto/system/rootfs  reservation                   none                          default
bto/system/rootfs  recordsize                    1M                            inherited from bto
bto/system/rootfs  mountpoint                    /                             local
bto/system/rootfs  sharenfs                      off                           default
bto/system/rootfs  checksum                      sha512                        inherited from bto
bto/system/rootfs  compression                   lz4                           inherited from bto
bto/system/rootfs  atime                         off                           inherited from bto
bto/system/rootfs  devices                       on                            default
bto/system/rootfs  exec                          on                            default
bto/system/rootfs  setuid                        on                            default
bto/system/rootfs  readonly                      off                           default
bto/system/rootfs  zoned                         off                           default
bto/system/rootfs  snapdir                       hidden                        default
bto/system/rootfs  aclinherit                    restricted                    default
bto/system/rootfs  createtxg                     47                            -
bto/system/rootfs  canmount                      on                            default
bto/system/rootfs  xattr                         sa                            inherited from bto
bto/system/rootfs  copies                        1                             default
bto/system/rootfs  version                       5                             -
bto/system/rootfs  utf8only                      on                            -
bto/system/rootfs  normalization                 formD                         -
bto/system/rootfs  casesensitivity               sensitive                     -
bto/system/rootfs  vscan                         off                           default
bto/system/rootfs  nbmand                        off                           default
bto/system/rootfs  sharesmb                      off                           default
bto/system/rootfs  refquota                      none                          default
bto/system/rootfs  refreservation                none                          default
bto/system/rootfs  guid                          10744756411378796574          -
bto/system/rootfs  primarycache                  all                           default
bto/system/rootfs  secondarycache                none                          inherited from bto
bto/system/rootfs  usedbysnapshots               12.5G                         -
bto/system/rootfs  usedbydataset                 4.68G                         -
bto/system/rootfs  usedbychildren                0B                            -
bto/system/rootfs  usedbyrefreservation          0B                            -
bto/system/rootfs  logbias                       latency                       default
bto/system/rootfs  objsetid                      394                           -
bto/system/rootfs  dedup                         off                           default
bto/system/rootfs  mlslabel                      none                          default
bto/system/rootfs  sync                          standard                      default
bto/system/rootfs  dnodesize                     auto                          inherited from bto
bto/system/rootfs  refcompressratio              2.09x                         -
bto/system/rootfs  written                       764M                          -
bto/system/rootfs  logicalused                   34.2G                         -
bto/system/rootfs  logicalreferenced             9.68G                         -
bto/system/rootfs  volmode                       default                       default
bto/system/rootfs  filesystem_limit              none                          default
bto/system/rootfs  snapshot_limit                none                          default
bto/system/rootfs  filesystem_count              none                          default
bto/system/rootfs  snapshot_count                none                          default
bto/system/rootfs  snapdev                       hidden                        default
bto/system/rootfs  acltype                       off                           default
bto/system/rootfs  context                       none                          default
bto/system/rootfs  fscontext                     none                          default
bto/system/rootfs  defcontext                    none                          default
bto/system/rootfs  rootcontext                   none                          default
bto/system/rootfs  relatime                      on                            temporary
bto/system/rootfs  redundant_metadata            all                           default
bto/system/rootfs  overlay                       off                           default
bto/system/rootfs  encryption                    off                           default
bto/system/rootfs  keylocation                   none                          default
bto/system/rootfs  keyformat                     none                          default
bto/system/rootfs  pbkdf2iters                   0                             default
bto/system/rootfs  special_small_blocks          0                             default

Describe the problem you're observing

After upgrading from zfs-0.8.1 to zfs-0.8.2 the snapshots under the root filesystem do not automount anymore. I have my root on zfs. Snapshots below other filesystems automount normally.
The snapshots can be mounted manually, so I have a workaround.

Describe how to reproduce the problem

  • Ensure your root filesystem is on zfs.
  • Upgrade to zfs and zfs-kmod-0.8.2 and reboot.
  • access any snapshot under /.zfs/snapshot/ and confirm that it's empty.

Include any warning/errors/backtraces from the system logs

I bisected it to commit 093bb64 in which case the following warning is logged in dmesg

 WARNING: Unable to automount (null)/.zfs/snapshot/20190101/bto/system/rootfs@20190101: 512
@behlendorf behlendorf added the Type: Defect Incorrect behavior (e.g. crash, hang) label Sep 30, 2019
@behlendorf
Copy link
Contributor

@janlam7 thank you for reporting this. Would it be possible for you to verify this small patch resolves the issue? Unfortunately, I don't have a test system handy which uses ZFS for the root filesystem.

diff --git a/module/os/linux/zfs/zfs_ctldir.c b/module/os/linux/zfs/zfs_ctldir.c
index 1e61ef0..3b2a6eb 100644
--- a/module/os/linux/zfs/zfs_ctldir.c
+++ b/module/os/linux/zfs/zfs_ctldir.c
@@ -1053,7 +1053,8 @@ zfsctl_snapshot_mount(struct path *path, int flags)
         * on mount.zfs(8).
         */
        snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
-           zfsvfs->z_vfs->vfs_mntpoint, dname(dentry));
+           zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
+           dname(dentry));
 
        /*
         * Multiple concurrent automounts of a snapshot are never allowed.

@janlam7
Copy link
Author

janlam7 commented Sep 30, 2019

@behlendorf I ended up applying the patch below, and indeed it solves the issue, nice! I changed the paths to match the 0.8-release branch.

diff --git a/module/zfs/zfs_ctldir.c b/module/zfs/zfs_ctldir.c
index 8acbbb61c..db583a647 100644
--- a/module/zfs/zfs_ctldir.c
+++ b/module/zfs/zfs_ctldir.c
@@ -1053,7 +1053,8 @@ zfsctl_snapshot_mount(struct path *path, int flags)
         * on mount.zfs(8).
         */
        snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
-           zfsvfs->z_vfs->vfs_mntpoint, dname(dentry));
+           zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
+            dname(dentry));
 
        /*
         * Multiple concurrent automounts of a snapshot are never allowed.

behlendorf added a commit to behlendorf/zfs that referenced this issue Sep 30, 2019
Commit 093bb64 resolved an automount failures for chroot'd processes
but inadvertently broke automounting for root filesystems where the
vfs_mntpoint is NULL.  Resolve the issue by checking for NULL in order
to generate the correct path.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#9381
@jhyeon
Copy link

jhyeon commented Oct 24, 2019

The above patch does not seem to be enough for me. Even with it applied to zfs-0.8.2, snapshots under /.zfs/snapshot/ are still empty.
The debug message from /proc/spl/kstat/zfs/dbgmsg is different from what is quoted above in that the path begins with /sysroot/ rather than (null)/. The same debug message appears with and without the patch after accessing a snapshot under /.zfs/snapshot/.
I am using linux kernel 5.2.20. I did not have this problem with zfs-0.8.1.

timestamp    message 
1571898386   spa.c:5623:spa_tryimport(): spa_tryimport: importing tank
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): LOADING
1571898386   vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/nvme0n1p3': best uberblock found for spa $import. txg 77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): LOADED
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): UNLOADING
1571898386   spa.c:5475:spa_import(): spa_import: importing tank
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config trusted): LOADING
1571898386   vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/nvme0n1p3': best uberblock found for spa tank. txg 77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=77982
1571898386   mmp.c:249:mmp_thread_start(): MMP thread started pool 'tank' gethrtime 3559400002
1571898386   spa.c:7577:spa_async_request(): spa=tank async request task=1
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config trusted): LOADED
1571898386   spa_history.c:319:spa_history_log_sync(): txg 77984 open pool version 5000; software version unknown; uts (none) 5.2.20-1 #1 SMP PREEMPT Thu Oct 10 00:15:30 KST 2019 x86_64
1571898386   spa.c:7577:spa_async_request(): spa=tank async request task=32
1571898386   spa_history.c:319:spa_history_log_sync(): txg 77986 import pool version 5000; software version unknown; uts (none) 5.2.20-1 #1 SMP PREEMPT Thu Oct 10 00:15:30 KST 2019 x86_64
1571898386   spa_history.c:306:spa_history_log_sync(): command: zpool import -N tank
1571898424   zfs_ctldir.c:1086:zfsctl_snapshot_mount(): Unable to automount /sysroot/.zfs/snapshot/zfs-auto-snap_daily-2019-10-22-0530 error=512

@jhyeon
Copy link

jhyeon commented Oct 24, 2019

I use initramfs created by dracut and openrc on the Funtoo distribution.

@hyegeek
Copy link

hyegeek commented Oct 27, 2019

This issue got me as well. I too am using an initrd (though a homegrown one and not dracut).

I can mount a snapshot with the following

mount -t zfs pool/root@snap /mnt

but the snapshot files under /.zfs/snapshots are all empty.

@Ryushin
Copy link

Ryushin commented Nov 26, 2019

Running Devuan Ascii. Patch worked fine against Debian's 0.8.2-3.

@jhyeon
Copy link

jhyeon commented Dec 3, 2019

@Ryushin do you use initramfs?

@Ryushin
Copy link

Ryushin commented Dec 3, 2019

@Ryushin do you use initramfs?

Yes, I use Sysvinit and initramfs. I also tested on a Debian Buster system with sysvinit and initramfs as well. So it worked on both Devuan and Debian.

@Ryushin
Copy link

Ryushin commented Dec 7, 2019

Seems I spoke too soon. Looks like I found a different kind of bug related to this. I've applied the patch listed above. My rpool looks like this:
rpool 1.27T 2.95T 176K none
rpool/ROOT 145G 2.95T 50.1G /
rpool/mysql 53.0G 2.95T 23.0G /var/lib/mysql
rpool/mysql-log 5.79G 2.95T 1.44G /var/lib/mysql-log
rpool/plexmediaserver 206G 2.95T 171G /var/lib/plexmediaserver
rpool/spool 115G 2.95T 35.5G /var/spool
rpool/virtual_machines 778G 2.95T 743G /var/lib/libvirt

I can see inside snapshots for rpool/ROOT:
windwalker:~# ls /.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/
bin boot.tar.xz dead.letter etc lib lost+found mnt opt proc run share sys tmp var
boot cdrom dev home lib64 media netshares path root sbin srv tftpboot usr

Looking inside snapshots for any other dataset in rpool results in "Too many levels of symbolic links" error:
ls /var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/
ls: cannot access '/var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/': Too many levels of symbolic links

tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Dec 26, 2019
Commit 093bb64 resolved an automount failures for chroot'd processes
but inadvertently broke automounting for root filesystems where the
vfs_mntpoint is NULL.  Resolve the issue by checking for NULL in order
to generate the correct path.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#9381
Closes openzfs#9384
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Dec 27, 2019
Commit 093bb64 resolved an automount failures for chroot'd processes
but inadvertently broke automounting for root filesystems where the
vfs_mntpoint is NULL.  Resolve the issue by checking for NULL in order
to generate the correct path.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#9381
Closes openzfs#9384
tonyhutter pushed a commit that referenced this issue Jan 23, 2020
Commit 093bb64 resolved an automount failures for chroot'd processes
but inadvertently broke automounting for root filesystems where the
vfs_mntpoint is NULL.  Resolve the issue by checking for NULL in order
to generate the correct path.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9381
Closes #9384
akorn added a commit to akorn/zfsbackup that referenced this issue Mar 2, 2020
 * Now supports zfs versions that have the snapshot automount bug
   (openzfs/zfs#9381), as well as filesystems
   that are not mounted or have legacy mountpoints.

 * The recursive snapshot feature handles (skips) child objects that are not
   filesystems (but e.g.  zvols).

 * We always leave the last successfully backed up snapshot (if any) intact,
   not the last snapshot as before.

 * No longer confused if the snapshot we're trying to create already exists
   (append numeric suffix to name).

 * Now supports recursive snapshots/backups of overlapping trees.

 * Share more code between recursive and nonrecursive snapshot case.

 * Switch to shared logging functions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

5 participants