Hang on listing snapshots and zpool history

[hjjg]

Hi,

I have two nodes with ZFS 0.6.3-2~trusty and I'm using zrep to replicate between both nodes. The kernel is 3.13.0-29-generic #53-Ubuntu SMP. The setup was fine for more than two weeks. This weekend my monitoring system reported timeouts getting information from ZFS. The following commands are hanging until killed (or CTRL-C if executed on the shell):

zfs get -Hp name quota used avail mountpoint type
zfs list -t snapshot
zpool history

Commands like zfs list, zpool list or zpool status are working. The dataset itself is exported with NFS and is usable. There is no debugging output in dmesg or syslog. I'm not using dedup. The pool has 49% free storage. I did not touch the pool at the weekend. It just replicated. There is no extra snapshot tool. zpool events has this two old entries:

Jun 17 2014 10:37:43.827943743 resource.fs.zfs.statechange
Jun 17 2014 10:37:44.015998285 ereport.fs.zfs.config.sync

I have not tried to reboot this machine. The second machine that I replicate to is usable and is able to list the available snapshots.

How do I get more debugging information? Is there a deadlock that is causing this problem?

Thanks in advance,

hjjg

[hjjg]

zfs list -t snapshot | wc -l
17650

Is there a recommandation for a tool to replicate the filesystem other than zrep?

[behlendorf]

To speed up zfs list when you have a large number of snapshots you can restrict the output to just the name. From the man page:

       zfs list [-r|-d depth] [-Hp] [-o property[,...]] [ -t type[,...]] [ -s property ] ... [
       -S property ] ... [filesystem|volume|snapshot] ...

           Lists the property information for the given datasets in tabular form. If specified,
           you can list property information by the absolute pathname or the relative pathname.
           By  default,  all file systems and volumes are displayed. Snapshots are displayed if
           the listsnaps property is on (the default is off). When listing  hundreds  or  thou-
           sands of snapshots performance can be improved by restricting the output to only the
           name.  In that case, it is recommended to use -o name -s name. The following  fields
           are displayed by default, name,used,available,referenced,mountpoint.

So something like this should speed it up considerably. You might be able to modify zrep to take advantage of this.

zfs list -t snap -o name -s name | wc -l