Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VERIFY3(0 == rt->rt_space) failed (0 == 95232), PANIC at range_tree.c:153:range_tree_destroy() #5953

Closed
runderwo opened this issue Apr 1, 2017 · 3 comments
Closed

VERIFY3(0 == rt->rt_space) failed (0 == 95232), PANIC at range_tree.c:153:range_tree_destroy() #5953

runderwo opened this issue Apr 1, 2017 · 3 comments
Labels
Status: Feedback requested More information is requested Status: Inactive Not being actively updated Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@runderwo
Copy link

runderwo commented Apr 1, 2017

Type Version/Name
Distribution Name Debian
Distribution Version stretch
Linux Kernel 4.6.0
Architecture amd64
ZFS Version 0.6.5.9-5
SPL Version 0.6.5.9-1

Describe the problem you're observing

I had an array with a disk with bad blocks that was preventing resilver from completing (it was restarting resilver endlessly for almost a month). To work around that, I was able to create a new zvol and copy the data out from the other zvol, since only the ext4 journal was damaged. I then deleted the damaged zvol, which allowed the resilver to complete (with errors in the nonexistent zvol, as seen below).

xanadu:~# zpool status -v
  pool: zfs0
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: resilvered 134G in 4h50m with 0 errors on Fri Mar 31 10:08:26 2017
config:

        NAME                          STATE     READ WRITE CKSUM
        zfs0                          ONLINE       0     0 7.52K
          raidz1-0                    ONLINE       0     0 30.1K
            zcrypt_WD-WCAVY2369232    ONLINE       0     0     0
            zcrypt_WD-WCAVY4923798    ONLINE       0     0     0
            spare-2                   ONLINE       0     0 12.7K
              zcrypt_WD-WCAVY4960321  ONLINE     216 2.03K     0
              zcrypt_YFJ067AA         ONLINE       0     0     0
            zcrypt_MN1220F32E00KD     ONLINE       0     0     0
        spares
          zcrypt_YFHZJ0TK             FAULTED   too many errors
          zcrypt_YFJ067AA             INUSE     currently in use

errors: Permanent errors have been detected in the following files:

        <0xffffffffffffffff>:<0x1>

Describe how to reproduce the problem

I am not sure how to reproduce it. A zpool export on the above array triggered the panic.

Include any warning/errors/backtraces from the system logs

xanadu:~# zpool export zfs0

Message from syslogd@xanadu at Mar 31 10:59:18 ...
 kernel:[4016244.343993] VERIFY3(0 == rt->rt_space) failed (0 == 95232)

Message from syslogd@xanadu at Mar 31 10:59:18 ...
 kernel:[4016244.344930] PANIC at range_tree.c:153:range_tree_destroy()

Screenshot of console attached

image
@runderwo
Copy link
Author

runderwo commented Apr 2, 2017 •
edited 

# zdb -mc zfs0 |head

Metaslabs:
        vdev          0
        metaslabs   116   offset                spacemap          free      
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap     38   free    49.6G
        metaslab      1   offset   1000000000   spacemap     59   free    53.7G
        metaslab      2   offset   2000000000   spacemap     60   free    48.5G
        metaslab      3   offset   3000000000   spacemap     62   free    42.3G
        metaslab      4   offset   4000000000   spacemap     63   free    41.8G
        metaslab      5   offset   5000000000   spacemap     64   free    13.4G
        metaslab      6   offset   6000000000   spacemap     65   free     401M
        metaslab      7   offset   7000000000   spacemap     66   free    13.1G
        metaslab      8   offset   8000000000   spacemap     67   free     372M
        metaslab      9   offset   9000000000   spacemap     68   free     274M
        metaslab     10   offset   a000000000   spacemap     69   free     437M
        metaslab     11   offset   b000000000   spacemap     72   free     288M
        metaslab     12   offset   c000000000   spacemap     74   free     710M
        metaslab     13   offset   d000000000   spacemap     75   free     846M
        metaslab     14   offset   e000000000   spacemap     76   free     229M
        metaslab     15   offset   f000000000   spacemap     77   free     670M
        metaslab     16   offset  10000000000   spacemap     78   free     217M
        metaslab     17   offset  11000000000   spacemap     79   free     237M
        metaslab     18   offset  12000000000   spacemap     80   free     678M
        metaslab     19   offset  13000000000   spacemap     81   free     527M
        metaslab     20   offset  14000000000   spacemap     84   free     541M
        metaslab     21   offset  15000000000   spacemap     86   free     435M
        metaslab     22   offset  16000000000   spacemap     37   free    50.0G
        metaslab     23   offset  17000000000   spacemap     87   free    6.56G
        metaslab     24   offset  18000000000   spacemap     88   free     615M
        metaslab     25   offset  19000000000   spacemap     61   free    64.0G
        metaslab     26   offset  1a000000000   spacemap     89   free    6.59G
        metaslab     27   offset  1b000000000   spacemap     90   free    51.3G
        metaslab     28   offset  1c000000000   spacemap     91   free      64G
        metaslab     29   offset  1d000000000   spacemap     92   free    64.0G
        metaslab     30   offset  1e000000000   spacemap     95   free    64.0G
        metaslab     31   offset  1f000000000   spacemap     96   free    64.0G
        metaslab     32   offset  20000000000   spacemap     97   free    62.3G
        metaslab     33   offset  21000000000   spacemap     71   free    2.87G
        metaslab     34   offset  22000000000   spacemap     73   free    3.59G
        metaslab     35   offset  23000000000   spacemap     98   free    63.3G
        metaslab     36   offset  24000000000   spacemap     99   free    63.3G
        metaslab     37   offset  25000000000   spacemap    100   free    7.99G
        metaslab     38   offset  26000000000   spacemap    127   free     834K
        metaslab     39   offset  27000000000   spacemap    128   free    19.9G
        metaslab     40   offset  28000000000   spacemap    130   free     692M
        metaslab     41   offset  29000000000   spacemap    131   free    1.72G
        metaslab     42   offset  2a000000000   spacemap     83   free    64.0G
        metaslab     43   offset  2b000000000   spacemap     85   free      64G
        metaslab     44   offset  2c000000000   spacemap     36   free    64.0G
        metaslab     45   offset  2d000000000   spacemap    132   free    15.4M
        metaslab     46   offset  2e000000000   spacemap    133   free    30.1G
        metaslab     47   offset  2f000000000   spacemap    149   free      64G
        metaslab     48   offset  30000000000   spacemap    150   free    64.0G
        metaslab     49   offset  31000000000   spacemap    151   free    64.0G
        metaslab     50   offset  32000000000   spacemap    154   free    64.0G
        metaslab     51   offset  33000000000   spacemap    155   free    64.0G
        metaslab     52   offset  34000000000   spacemap     94   free     702M
        metaslab     53   offset  35000000000   spacemap    156   free    64.0G
        metaslab     54   offset  36000000000   spacemap    157   free    46.4G
        metaslab     55   offset  37000000000   spacemap     70   free    63.3G
        metaslab     56   offset  38000000000   spacemap    158   free    1.12M
        metaslab     57   offset  39000000000   spacemap    159   free    1.49M
        metaslab     58   offset  3a000000000   spacemap    160   free    67.7M
        metaslab     59   offset  3b000000000   spacemap    161   free    4.49M
        metaslab     60   offset  3c000000000   spacemap    126   free    63.0G
        metaslab     61   offset  3d000000000   spacemap    163   free    1.06M
        metaslab     62   offset  3e000000000   spacemap    129   free    63.2G
        metaslab     63   offset  3f000000000   spacemap    164   free    17.2G
        metaslab     64   offset  40000000000   spacemap     82   free    63.8G
        metaslab     65   offset  41000000000   spacemap    102   free     101M
        metaslab     66   offset  42000000000   spacemap      0   free      64G
        metaslab     67   offset  43000000000   spacemap      0   free      64G
        metaslab     68   offset  44000000000   spacemap      0   free      64G
        metaslab     69   offset  45000000000   spacemap      0   free      64G
        metaslab     70   offset  46000000000   spacemap      0   free      64G
        metaslab     71   offset  47000000000   spacemap      0   free      64G
        metaslab     72   offset  48000000000   spacemap    153   free    63.3G
        metaslab     73   offset  49000000000   spacemap      0   free      64G
        metaslab     74   offset  4a000000000   spacemap     93   free    62.8G
        metaslab     75   offset  4b000000000   spacemap      0   free      64G
        metaslab     76   offset  4c000000000   spacemap      0   free      64G
        metaslab     77   offset  4d000000000   spacemap      0   free      64G
        metaslab     78   offset  4e000000000   spacemap      0   free      64G
        metaslab     79   offset  4f000000000   spacemap      0   free      64G
        metaslab     80   offset  50000000000   spacemap      0   free      64G
        metaslab     81   offset  51000000000   spacemap      0   free      64G
        metaslab     82   offset  52000000000   spacemap    125   free    63.2G
        metaslab     83   offset  53000000000   spacemap      0   free      64G
        metaslab     84   offset  54000000000   spacemap      0   free      64G
        metaslab     85   offset  55000000000   spacemap      0   free      64G
        metaslab     86   offset  56000000000   spacemap      0   free      64G
        metaslab     87   offset  57000000000   spacemap      0   free      64G
        metaslab     88   offset  58000000000   spacemap      0   free      64G
        metaslab     89   offset  59000000000   spacemap      0   free      64G
        metaslab     90   offset  5a000000000   spacemap      0   free      64G
        metaslab     91   offset  5b000000000   spacemap      0   free      64G
        metaslab     92   offset  5c000000000   spacemap      0   free      64G
        metaslab     93   offset  5d000000000   spacemap      0   free      64G
        metaslab     94   offset  5e000000000   spacemap    152   free    64.0G
        metaslab     95   offset  5f000000000   spacemap      0   free      64G
        metaslab     96   offset  60000000000   spacemap      0   free      64G
        metaslab     97   offset  61000000000   spacemap      0   free      64G
        metaslab     98   offset  62000000000   spacemap      0   free      64G
        metaslab     99   offset  63000000000   spacemap      0   free      64G
        metaslab    100   offset  64000000000   spacemap      0   free      64G
        metaslab    101   offset  65000000000   spacemap      0   free      64G
        metaslab    102   offset  66000000000   spacemap      0   free      64G
        metaslab    103   offset  67000000000   spacemap      0   free      64G
        metaslab    104   offset  68000000000   spacemap    162   free    64.0G
        metaslab    105   offset  69000000000   spacemap      0   free      64G
        metaslab    106   offset  6a000000000   spacemap      0   free      64G
        metaslab    107   offset  6b000000000   spacemap      0   free      64G
        metaslab    108   offset  6c000000000   spacemap      0   free      64G
        metaslab    109   offset  6d000000000   spacemap      0   free      64G
        metaslab    110   offset  6e000000000   spacemap      0   free      64G
        metaslab    111   offset  6f000000000   spacemap      0   free      64G
        metaslab    112   offset  70000000000   spacemap      0   free      64G
        metaslab    113   offset  71000000000   spacemap      0   free      64G
        metaslab    114   offset  72000000000   spacemap      0   free      64G
        metaslab    115   offset  73000000000   spacemap      0   free      64G


Traversing all blocks to verify metadata checksums and verify nothing leaked ...

loading space map for vdev 0 of 1, metaslab 104 of 116 ...
zdb: zdb.c:2768: Assertion `0 == bptree_iterate(spa->spa_meta_objset, spa->spa_dsl_pool->dp_bptree_obj, B_FALSE, count_block_cb, &zcb, ((void *)0)) (0x0 == 0x34)' failed.
Aborted

@org87
Copy link

org87 commented Aug 28, 2017

I had the same problem. kernel is 4.4.66 and zfs/spl is 0.6.5.8-1.
[428695.428349] VERIFY3(0 == rt->rt_space) failed (0 == 730112)
[428695.429356] PANIC at range_tree.c:153:range_tree_destroy()
[428695.430331] Showing stack for process 14675
[428695.430337] CPU: 8 PID: 14675 Comm: zpool Tainted: P O 4.4.66-v64 #1
[428695.430339] Hardware name: AIC HA401-LB2/LIBRA, BIOS LIBAS061 06/28/2016
[428695.430342] 0000000000000000 0000000000000099 ffffffff812ad192 0000000000000000
[428695.430348] ffffffff8165b340 0000000000000099 ffff88102648bb78 ffffffffa04a3de0
[428695.430352] ffffffffa03a8987 ffff88107e3df920 ffffffffa04b80f0 2833594649524556
[428695.430355] Call Trace:
[428695.430372] [] ? dump_stack+0x4a/0x68
[428695.430383] [] ? spl_panic+0xb7/0xf0 [spl]
[428695.430390] [] ? __cache_free_alien+0xaf/0x130
[428695.430393] [] ? cache_flusharray+0xbe/0x120
[428695.430441] [] ? range_tree_destroy+0x5a/0x60 [zfs]
[428695.430471] [] ? metaslab_fini+0xff/0x1a0 [zfs]
[428695.430502] [] ? vdev_metaslab_fini+0x40/0x80 [zfs]
[428695.430530] [] ? vdev_free+0x57/0x260 [zfs]
[428695.430558] [] ? vdev_free+0x39/0x260 [zfs]
[428695.430590] [] ? spa_unload+0xb7/0x250 [zfs]
[428695.430623] [] ? spa_export_common+0x25b/0x3b0 [zfs]
[428695.430651] [] ? zfsdev_ioctl+0x486/0x4c0 [zfs]
[428695.430661] [] ? do_vfs_ioctl+0x85/0x4b0
[428695.430669] [] ? __do_page_fault+0x1c4/0x450
[428695.430674] [] ? SyS_ioctl+0x98/0xa0
[428695.430680] [] ? entry_SYSCALL_64_fastpath+0x12/0x6a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Feedback requested More information is requested Status: Inactive Not being actively updated Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@runderwo @org87 and others