Permanent errors, <0xffffffffffffffff>:<0x0>, due to my own stupid mistake.

[gnif]

I have a zfs pool I made an error with and screwed up, I had just rebuilt my local server and when I when to move the running machine the SATA power cable moved enough to disconnect two of the three disks that contain my ZFS pool. This obvioulsy caused a ton of errors, but thankfully the array was not very active at the time.

Once the physical issue was fixed, one disk was reading as requring a resilver, and the array was in the degraded state, but working. A scrub reported some file errors in logs, no big deal there, but it also reported an error with one of the pools in the form of: datapool/pool<0:0>

This pool had a ton of errors, but found the most recent snapshot was intact, so I reverted to it which corrected things. I also performed a resilver to add back the extra disk and then another scrub which now states the pool is in the ONLINE state... but with one error:

errors: Permanent errors have been detected in the following files:

        <0xffffffffffffffff>:<0x0>

I do not have enough storage to backup the data to recreate the pool, nor can I afford to buy additional disks at current. This seems related to #3094, but since it was caused by a faulty connector I am not sure if any of it applies.

Any help here would be greatly appreciated.

Here is the output of zdb -mc:

Metaslabs:
        vdev          0
        metaslabs   174   offset                spacemap          free
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap    240   free    54.1G
        metaslab      1   offset   1000000000   spacemap     35   free    26.3G
        metaslab      2   offset   2000000000   spacemap     59   free    6.18G
        metaslab      3   offset   3000000000   spacemap    241   free     560M
        metaslab      4   offset   4000000000   spacemap     36   free    5.66G
        metaslab      5   offset   5000000000   spacemap     60   free    34.8G
        metaslab      6   offset   6000000000   spacemap     61   free     455M
        metaslab      7   offset   7000000000   spacemap     62   free    3.29G
        metaslab      8   offset   8000000000   spacemap     37   free    28.7G
        metaslab      9   offset   9000000000   spacemap    252   free    8.23G
        metaslab     10   offset   a000000000   spacemap    288   free    7.52G
        metaslab     11   offset   b000000000   spacemap    281   free    17.7G
        metaslab     12   offset   c000000000   spacemap     38   free    20.0G
        metaslab     13   offset   d000000000   spacemap    300   free    19.6G
        metaslab     14   offset   e000000000   spacemap    309   free    71.9M
        metaslab     15   offset   f000000000   spacemap    256   free    10.4G
        metaslab     16   offset  10000000000   spacemap     63   free     168M
        metaslab     17   offset  11000000000   spacemap     64   free    8.10G
        metaslab     18   offset  12000000000   spacemap     65   free     9.6G
        metaslab     19   offset  13000000000   spacemap     40   free     282M
        metaslab     20   offset  14000000000   spacemap     66   free    86.5M
        metaslab     21   offset  15000000000   spacemap     67   free    6.59G
        metaslab     22   offset  16000000000   spacemap     68   free    16.6G
        metaslab     23   offset  17000000000   spacemap    283   free    2.80G
        metaslab     24   offset  18000000000   spacemap    287   free    1.94G
        metaslab     25   offset  19000000000   spacemap    304   free    70.6M
        metaslab     26   offset  1a000000000   spacemap    313   free    18.8M
        metaslab     27   offset  1b000000000   spacemap     69   free    8.63G
        metaslab     28   offset  1c000000000   spacemap     74   free    41.2M
        metaslab     29   offset  1d000000000   spacemap    239   free    64.0G
        metaslab     30   offset  1e000000000   spacemap    251   free    1.09G
        metaslab     31   offset  1f000000000   spacemap     79   free    51.2M
        metaslab     32   offset  20000000000   spacemap     80   free    89.3M
        metaslab     33   offset  21000000000   spacemap     81   free     479M
        metaslab     34   offset  22000000000   spacemap     82   free    20.6G
        metaslab     35   offset  23000000000   spacemap    310   free    44.6M
        metaslab     36   offset  24000000000   spacemap     83   free    7.22G
        metaslab     37   offset  25000000000   spacemap     84   free    11.0G
        metaslab     38   offset  26000000000   spacemap     85   free     433M
        metaslab     39   offset  27000000000   spacemap     86   free     508M
        metaslab     40   offset  28000000000   spacemap    254   free    33.4G
        metaslab     41   offset  29000000000   spacemap     93   free    30.5G
        metaslab     42   offset  2a000000000   spacemap     94   free    9.13G
        metaslab     43   offset  2b000000000   spacemap     95   free    41.5M
        metaslab     44   offset  2c000000000   spacemap     97   free    7.26G
        metaslab     45   offset  2d000000000   spacemap    106   free     200M
        metaslab     46   offset  2e000000000   spacemap    305   free    38.7M
        metaslab     47   offset  2f000000000   spacemap    107   free    12.7M
        metaslab     48   offset  30000000000   spacemap    108   free     173M
        metaslab     49   offset  31000000000   spacemap    116   free    15.7G
        metaslab     50   offset  32000000000   spacemap    117   free     131M
        metaslab     51   offset  33000000000   spacemap    122   free    39.8M
        metaslab     52   offset  34000000000   spacemap    123   free     131M
        metaslab     53   offset  35000000000   spacemap    303   free    11.1G
        metaslab     54   offset  36000000000   spacemap    124   free    20.5G
        metaslab     55   offset  37000000000   spacemap    301   free    49.8M
        metaslab     56   offset  38000000000   spacemap    311   free    6.28G
        metaslab     57   offset  39000000000   spacemap    125   free    10.4G
        metaslab     58   offset  3a000000000   spacemap    238   free    16.0G
        metaslab     59   offset  3b000000000   spacemap    126   free    42.0M
        metaslab     60   offset  3c000000000   spacemap    146   free     298M
        metaslab     61   offset  3d000000000   spacemap    165   free    3.98G
        metaslab     62   offset  3e000000000   spacemap     41   free     253M
        metaslab     63   offset  3f000000000   spacemap    266   free     359M
        metaslab     64   offset  40000000000   spacemap    261   free    7.74G
        metaslab     65   offset  41000000000   spacemap    279   free    34.9G
        metaslab     66   offset  42000000000   spacemap    179   free    6.21G
        metaslab     67   offset  43000000000   spacemap    291   free    49.3G
        metaslab     68   offset  44000000000   spacemap    246   free    59.6G
        metaslab     69   offset  45000000000   spacemap    253   free     505M
        metaslab     70   offset  46000000000   spacemap    307   free    25.3G
        metaslab     71   offset  47000000000   spacemap    269   free    16.2G
        metaslab     72   offset  48000000000   spacemap    232   free    56.8G
        metaslab     73   offset  49000000000   spacemap    276   free    2.12G
        metaslab     74   offset  4a000000000   spacemap    257   free    11.4G
        metaslab     75   offset  4b000000000   spacemap    306   free    77.4M
        metaslab     76   offset  4c000000000   spacemap    181   free    20.8G
        metaslab     77   offset  4d000000000   spacemap    290   free     243M
        metaslab     78   offset  4e000000000   spacemap    299   free    1.07G
        metaslab     79   offset  4f000000000   spacemap    280   free    69.1M
        metaslab     80   offset  50000000000   spacemap    295   free    2.81G
        metaslab     81   offset  51000000000   spacemap    292   free    2.23G
        metaslab     82   offset  52000000000   spacemap    302   free    22.1G
        metaslab     83   offset  53000000000   spacemap    293   free    1.73G
        metaslab     84   offset  54000000000   spacemap    259   free    1.50G
        metaslab     85   offset  55000000000   spacemap    264   free    43.9G
        metaslab     86   offset  56000000000   spacemap    294   free     911M
        metaslab     87   offset  57000000000   spacemap    262   free    67.2M
        metaslab     88   offset  58000000000   spacemap    277   free    15.9G
        metaslab     89   offset  59000000000   spacemap    296   free     903M
        metaslab     90   offset  5a000000000   spacemap    237   free    48.6G
        metaslab     91   offset  5b000000000   spacemap    243   free    44.7G
        metaslab     92   offset  5c000000000   spacemap    250   free    1.30G
        metaslab     93   offset  5d000000000   spacemap    248   free    20.1G
        metaslab     94   offset  5e000000000   spacemap    297   free    3.25G
        metaslab     95   offset  5f000000000   spacemap    245   free    17.2G
        metaslab     96   offset  60000000000   spacemap    298   free    24.9G
        metaslab     97   offset  61000000000   spacemap    278   free     591M
        metaslab     98   offset  62000000000   spacemap    282   free    24.4G
        metaslab     99   offset  63000000000   spacemap    285   free    13.8G
        metaslab    100   offset  64000000000   spacemap    234   free    42.4G
        metaslab    101   offset  65000000000   spacemap    249   free    2.93G
        metaslab    102   offset  66000000000   spacemap    268   free    1.75G
        metaslab    103   offset  67000000000   spacemap    247   free    64.0G
        metaslab    104   offset  68000000000   spacemap     50   free    64.0G
        metaslab    105   offset  69000000000   spacemap    101   free    63.9G
        metaslab    106   offset  6a000000000   spacemap    113   free    59.2G
        metaslab    107   offset  6b000000000   spacemap    255   free    4.60G
        metaslab    108   offset  6c000000000   spacemap    391   free    17.6G
        metaslab    109   offset  6d000000000   spacemap     78   free    61.3G
        metaslab    110   offset  6e000000000   spacemap    392   free    38.4G
        metaslab    111   offset  6f000000000   spacemap    312   free    1.02G
        metaslab    112   offset  70000000000   spacemap    236   free     782M
        metaslab    113   offset  71000000000   spacemap    393   free     538M
        metaslab    114   offset  72000000000   spacemap    395   free    6.50G
        metaslab    115   offset  73000000000   spacemap    396   free    54.1G
        metaslab    116   offset  74000000000   spacemap    289   free    30.4G
        metaslab    117   offset  75000000000   spacemap    350   free    56.2G
        metaslab    118   offset  76000000000   spacemap     48   free    53.5G
        metaslab    119   offset  77000000000   spacemap     52   free    4.34G
        metaslab    120   offset  78000000000   spacemap     71   free    22.8G
        metaslab    121   offset  79000000000   spacemap    355   free    63.3M
        metaslab    122   offset  7a000000000   spacemap    356   free    67.6M
        metaslab    123   offset  7b000000000   spacemap    357   free    45.1M
        metaslab    124   offset  7c000000000   spacemap    244   free    63.3G
        metaslab    125   offset  7d000000000   spacemap    242   free    4.47G
        metaslab    126   offset  7e000000000   spacemap    233   free    16.4G
        metaslab    127   offset  7f000000000   spacemap    358   free    7.56G
        metaslab    128   offset  80000000000   spacemap    359   free    34.5M
        metaslab    129   offset  81000000000   spacemap    360   free    25.5G
        metaslab    130   offset  82000000000   spacemap    398   free    98.5M
        metaslab    131   offset  83000000000   spacemap    399   free    20.3G
        metaslab    132   offset  84000000000   spacemap   3658   free    21.3G
        metaslab    133   offset  85000000000   spacemap     70   free    14.3G
        metaslab    134   offset  86000000000   spacemap   4883   free    25.2G
        metaslab    135   offset  87000000000   spacemap   5387   free    23.4G
        metaslab    136   offset  88000000000   spacemap   5960   free     9.5G
        metaslab    137   offset  89000000000   spacemap   6428   free    26.8G
        metaslab    138   offset  8a000000000   spacemap    100   free    48.4G
        metaslab    139   offset  8b000000000   spacemap    112   free    48.5G
        metaslab    140   offset  8c000000000   spacemap    235   free     9.6G
        metaslab    141   offset  8d000000000   spacemap     46   free    16.6G
        metaslab    142   offset  8e000000000   spacemap    450   free    44.8G
        metaslab    143   offset  8f000000000   spacemap   8195   free    35.3G
        metaslab    144   offset  90000000000   spacemap   9166   free    48.8G
        metaslab    145   offset  91000000000   spacemap   4984   free    44.1G
        metaslab    146   offset  92000000000   spacemap     58   free    4.14G
        metaslab    147   offset  93000000000   spacemap    560   free    2.67G
        metaslab    148   offset  94000000000   spacemap   1289   free     139M
        metaslab    149   offset  95000000000   spacemap    613   free    23.8G
        metaslab    150   offset  96000000000   spacemap     57   free    29.5G
        metaslab    151   offset  97000000000   spacemap     72   free    19.3G
        metaslab    152   offset  98000000000   spacemap     49   free    1.55G
        metaslab    153   offset  99000000000   spacemap   2305   free    3.26G
        metaslab    154   offset  9a000000000   spacemap    354   free    5.18G
        metaslab    155   offset  9b000000000   spacemap    694   free    48.1G
        metaslab    156   offset  9c000000000   spacemap     87   free    62.2G
        metaslab    157   offset  9d000000000   spacemap    786   free    6.87G
        metaslab    158   offset  9e000000000   spacemap   1224   free    43.4G
        metaslab    159   offset  9f000000000   spacemap    614   free    63.2G
        metaslab    160   offset  a0000000000   spacemap     88   free    2.04G
        metaslab    161   offset  a1000000000   spacemap    622   free    2.89G
        metaslab    162   offset  a2000000000   spacemap   1484   free    25.0G
        metaslab    163   offset  a3000000000   spacemap   2273   free    60.6G
        metaslab    164   offset  a4000000000   spacemap     90   free    3.13G
        metaslab    165   offset  a5000000000   spacemap   1263   free     138M
        metaslab    166   offset  a6000000000   spacemap   6804   free    64.0G
        metaslab    167   offset  a7000000000   spacemap   4882   free    63.9G
        metaslab    168   offset  a8000000000   spacemap    879   free    29.4G
        metaslab    169   offset  a9000000000   spacemap     23   free    64.0G
        metaslab    170   offset  aa000000000   spacemap    115   free    64.0G
        metaslab    171   offset  ab000000000   spacemap    384   free    64.0G
        metaslab    172   offset  ac000000000   spacemap      0   free      64G
        metaslab    173   offset  ad000000000   spacemap      0   free      64G

        vdev          1
        metaslabs    63   offset                spacemap          free
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap    133   free      16M
        metaslab      1   offset      1000000   spacemap    505   free      16M
        metaslab      2   offset      2000000   spacemap    636   free      16M
        metaslab      3   offset      3000000   spacemap    635   free    15.9M
        metaslab      4   offset      4000000   spacemap    634   free      16M
        metaslab      5   offset      5000000   spacemap    633   free      16M
        metaslab      6   offset      6000000   spacemap    632   free      16M
        metaslab      7   offset      7000000   spacemap    631   free      16M
        metaslab      8   offset      8000000   spacemap    630   free    16.0M
        metaslab      9   offset      9000000   spacemap    629   free    15.9M
        metaslab     10   offset      a000000   spacemap    628   free      16M
        metaslab     11   offset      b000000   spacemap    627   free      16M
        metaslab     12   offset      c000000   spacemap    626   free      16M
        metaslab     13   offset      d000000   spacemap    625   free      16M
        metaslab     14   offset      e000000   spacemap    624   free      16M
        metaslab     15   offset      f000000   spacemap    623   free      16M
        metaslab     16   offset     10000000   spacemap    621   free      16M
        metaslab     17   offset     11000000   spacemap    620   free      16M
        metaslab     18   offset     12000000   spacemap    619   free      16M
        metaslab     19   offset     13000000   spacemap    618   free      16M
        metaslab     20   offset     14000000   spacemap    617   free    15.8M
        metaslab     21   offset     15000000   spacemap    616   free      16M
        metaslab     22   offset     16000000   spacemap    615   free      16M
        metaslab     23   offset     17000000   spacemap    612   free      16M
        metaslab     24   offset     18000000   spacemap    611   free      16M
        metaslab     25   offset     19000000   spacemap    610   free      16M
        metaslab     26   offset     1a000000   spacemap    609   free      16M
        metaslab     27   offset     1b000000   spacemap    608   free      16M
        metaslab     28   offset     1c000000   spacemap    607   free      16M
        metaslab     29   offset     1d000000   spacemap    606   free      16M
        metaslab     30   offset     1e000000   spacemap    605   free      16M
        metaslab     31   offset     1f000000   spacemap    604   free      16M
        metaslab     32   offset     20000000   spacemap    603   free      16M
        metaslab     33   offset     21000000   spacemap    602   free      16M
        metaslab     34   offset     22000000   spacemap    601   free      16M
        metaslab     35   offset     23000000   spacemap    600   free      16M
        metaslab     36   offset     24000000   spacemap    599   free      16M
        metaslab     37   offset     25000000   spacemap    598   free      16M
        metaslab     38   offset     26000000   spacemap    597   free      16M
        metaslab     39   offset     27000000   spacemap    596   free      16M
        metaslab     40   offset     28000000   spacemap    595   free      16M
        metaslab     41   offset     29000000   spacemap    594   free      16M
        metaslab     42   offset     2a000000   spacemap    593   free      16M
        metaslab     43   offset     2b000000   spacemap    592   free      16M
        metaslab     44   offset     2c000000   spacemap    591   free      16M
        metaslab     45   offset     2d000000   spacemap    590   free      16M
        metaslab     46   offset     2e000000   spacemap    589   free      16M
        metaslab     47   offset     2f000000   spacemap    588   free      16M
        metaslab     48   offset     30000000   spacemap    587   free      16M
        metaslab     49   offset     31000000   spacemap    586   free      16M
        metaslab     50   offset     32000000   spacemap    585   free      16M
        metaslab     51   offset     33000000   spacemap    584   free      16M
        metaslab     52   offset     34000000   spacemap    583   free      16M
        metaslab     53   offset     35000000   spacemap    582   free      16M
        metaslab     54   offset     36000000   spacemap    581   free      16M
        metaslab     55   offset     37000000   spacemap    580   free      16M
        metaslab     56   offset     38000000   spacemap    579   free      16M
        metaslab     57   offset     39000000   spacemap    578   free      16M
        metaslab     58   offset     3a000000   spacemap    577   free      16M
        metaslab     59   offset     3b000000   spacemap    576   free      16M
        metaslab     60   offset     3c000000   spacemap    575   free      16M
        metaslab     61   offset     3d000000   spacemap    574   free      16M
        metaslab     62   offset     3e000000   spacemap    573   free      16M

        vdev          2
        metaslabs    63   offset                spacemap          free
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap    139   free      16M
        metaslab      1   offset      1000000   spacemap    504   free      16M
        metaslab      2   offset      2000000   spacemap    572   free      16M
        metaslab      3   offset      3000000   spacemap    571   free      16M
        metaslab      4   offset      4000000   spacemap    570   free      16M
        metaslab      5   offset      5000000   spacemap    569   free      16M
        metaslab      6   offset      6000000   spacemap    568   free    16.0M
        metaslab      7   offset      7000000   spacemap    567   free      16M
        metaslab      8   offset      8000000   spacemap    566   free      16M
        metaslab      9   offset      9000000   spacemap    565   free      16M
        metaslab     10   offset      a000000   spacemap    564   free      16M
        metaslab     11   offset      b000000   spacemap    563   free      16M
        metaslab     12   offset      c000000   spacemap    562   free      16M
        metaslab     13   offset      d000000   spacemap    561   free      16M
        metaslab     14   offset      e000000   spacemap    559   free      16M
        metaslab     15   offset      f000000   spacemap    558   free    16.0M
        metaslab     16   offset     10000000   spacemap    557   free      16M
        metaslab     17   offset     11000000   spacemap    556   free      16M
        metaslab     18   offset     12000000   spacemap    555   free      16M
        metaslab     19   offset     13000000   spacemap    554   free      16M
        metaslab     20   offset     14000000   spacemap    553   free      16M
        metaslab     21   offset     15000000   spacemap    552   free      16M
        metaslab     22   offset     16000000   spacemap    551   free      16M
        metaslab     23   offset     17000000   spacemap    550   free      16M
        metaslab     24   offset     18000000   spacemap    549   free      16M
        metaslab     25   offset     19000000   spacemap    548   free      16M
        metaslab     26   offset     1a000000   spacemap    547   free      16M
        metaslab     27   offset     1b000000   spacemap    546   free      16M
        metaslab     28   offset     1c000000   spacemap    540   free      16M
        metaslab     29   offset     1d000000   spacemap    539   free      16M
        metaslab     30   offset     1e000000   spacemap    538   free      16M
        metaslab     31   offset     1f000000   spacemap    537   free      16M
        metaslab     32   offset     20000000   spacemap    536   free      16M
        metaslab     33   offset     21000000   spacemap    535   free      16M
        metaslab     34   offset     22000000   spacemap    534   free      16M
        metaslab     35   offset     23000000   spacemap    533   free      16M
        metaslab     36   offset     24000000   spacemap    532   free      16M
        metaslab     37   offset     25000000   spacemap    531   free      16M
        metaslab     38   offset     26000000   spacemap    530   free      16M
        metaslab     39   offset     27000000   spacemap    529   free    16.0M
        metaslab     40   offset     28000000   spacemap    528   free      16M
        metaslab     41   offset     29000000   spacemap    527   free      16M
        metaslab     42   offset     2a000000   spacemap    526   free      16M
        metaslab     43   offset     2b000000   spacemap    525   free      16M
        metaslab     44   offset     2c000000   spacemap    524   free      16M
        metaslab     45   offset     2d000000   spacemap    523   free      16M
        metaslab     46   offset     2e000000   spacemap    522   free      16M
        metaslab     47   offset     2f000000   spacemap    521   free      16M
        metaslab     48   offset     30000000   spacemap    520   free      16M
        metaslab     49   offset     31000000   spacemap    519   free      16M
        metaslab     50   offset     32000000   spacemap    518   free      16M
        metaslab     51   offset     33000000   spacemap    517   free      16M
        metaslab     52   offset     34000000   spacemap    516   free      16M
        metaslab     53   offset     35000000   spacemap    515   free      16M
        metaslab     54   offset     36000000   spacemap    514   free      16M
        metaslab     55   offset     37000000   spacemap    513   free      16M
        metaslab     56   offset     38000000   spacemap    512   free      16M
        metaslab     57   offset     39000000   spacemap    511   free      16M
        metaslab     58   offset     3a000000   spacemap    510   free      16M
        metaslab     59   offset     3b000000   spacemap    509   free      16M
        metaslab     60   offset     3c000000   spacemap    508   free      16M
        metaslab     61   offset     3d000000   spacemap    507   free      16M
        metaslab     62   offset     3e000000   spacemap    506   free      16M


Traversing all blocks to verify metadata checksums and verify nothing leaked ...

loading space map for vdev 2 of 3, metaslab 62 of 63 .....
zdb: zdb.c:2759: Assertion `0 == bptree_iterate(spa->spa_meta_objset, spa->spa_dsl_pool->dp_bptree_obj, B_FALSE, count_block_cb, &zcb, ((void *)0)) (0x0 == 0x34)' failed.
Aborted

[gnif]

Closing as this issue is ancient and I have since moved to a new pool, and since moved away from ZFS due to lack of recovery tools.