Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FreeBSD 13, after importing pool system panics #14973

Open
nbari opened this issue Jun 11, 2023 · 0 comments
Open

FreeBSD 13, after importing pool system panics #14973

nbari opened this issue Jun 11, 2023 · 0 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@nbari
Copy link

nbari commented Jun 11, 2023

System information

FreeBSD - 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64

$ zfs version
zfs-2.1.9-FreeBSD_g92e0d9d18
zfs-kmod-2.1.9-FreeBSD_g92e0d9d18

Describe the problem you're observing

The system started to reboot randomly:

In an attempt to fix it, I notice that swapping the SATA cables of the disk fixed the problem temporally (indeed to pool became healthy):

# zpool status -x
all pools are healthy

Describe how to reproduce the problem

After some weeks, the problem started to happen again, this time I reinstalled the OS (latest FreeBSD 13-2) but now added a SWAP partition of the same amount of memory (128GB)

Using the default of 2GB wasn't enough to catch the panic

After booting, I mount the pool:

zpool import tank

And started to write/read from it, that creates the panic, but if the pool is mounted in read-only mode there is no panic/reboot

Include any warning/errors/backtraces from the system logs

output of: /var/crash/info.0

Dump header from device: /dev/nvd0p3
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 4681056256
  Blocksize: 512
  Compression: none
  Dumptime: 2023-06-11 11:44:47 +0000
  Hostname: home
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC
  Panic String: Solaris(panic): zfs: attempting to increase fill beyond max; probable double add in segment [0:787f7000]
  Dump Parity: 2582495081
  Bounds: 0
  Dump Status: good

Output of kgdb /boot/kernel/kernel /var/crash/vmcore.0

panic: Solaris(panic): zfs: attempting to increase fill beyond max; probable double add in segment [0:787f7000]
cpuid = 20
time = 1686483887
KDB: stack backtrace:
#0 0xffffffff80c53dc5 at kdb_backtrace+0x65
#1 0xffffffff80c06741 at vpanic+0x151
#2 0xffffffff80c065e3 at panic+0x43
#3 0xffffffff82164bcb at vcmn_err+0xeb
#4 0xffffffff8224f549 at zfs_panic_recover+0x59
#5 0xffffffff8222b10a at range_tree_adjust_fill+0x29a
#6 0xffffffff8222b5a4 at range_tree_add_impl+0x204
#7 0xffffffff82216517 at scan_io_queue_insert_impl+0xa7
#8 0xffffffff82215b43 at dsl_scan_scrub_cb+0xa63
#9 0xffffffff82217f3c at dsl_scan_visitbp+0x4cc
#10 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#11 0xffffffff822180f3 at dsl_scan_visitbp+0x683
#12 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#13 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#14 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#15 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#16 0xffffffff82217e96 at dsl_scan_visitbp+0x426
#17 0xffffffff82217dae at dsl_scan_visitbp+0x33e
Uptime: 5m11s
Dumping 4464 out of 130941 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Backtrace:

(kgdb) backtrace
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396
#2  0xffffffff80c0630a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484
#3  0xffffffff80c067ae in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe024d4dadc0) at /usr/src/sys/kern/kern_shutdown.c:923
#4  0xffffffff80c065e3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847
#5  0xffffffff82164bcb in vcmn_err (ce=<optimized out>,
    fmt=0xffffffff82397dc1 "zfs: attempting to increase fill beyond max; probable double add in segment [%llx:%llx]", adx=0xfffffe024d4daf50)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_cmn_err.c:60
#6  0xffffffff8224f549 in zfs_panic_recover (fmt=<unavailable>) at /usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c:1566
#7  0xffffffff8222b10a in range_tree_adjust_fill (rt=0xfffff80148a17400, rs=0xfffff801f79e2010, delta=131072) at /usr/src/sys/contrib/openzfs/module/zfs/range_tree.c:261
#8  0xffffffff8222b5a4 in range_tree_add_impl (arg=0xfffff80148a17400, start=174874624, size=<optimized out>, fill=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/zfs/range_tree.c:311
#9  0xffffffff8222b39a in range_tree_add (arg=<unavailable>, start=<unavailable>, size=<unavailable>) at /usr/src/sys/contrib/openzfs/module/zfs/range_tree.c:422
#10 0xffffffff82216517 in scan_io_queue_insert_impl (queue=queue@entry=0xfffff80148d15900, sio=0xfffff802a861abb0)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:3872
#11 0xffffffff82215b43 in scan_io_queue_insert (queue=0xfffff80148d15900, bp=0xfffffe0256bd9000, zio_flags=25166000, zb=<unavailable>, dva_i=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:3896
#12 dsl_scan_enqueue (dp=<optimized out>, bp=0xfffffe0256bd9000, zio_flags=25166000, zb=<unavailable>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:3933
#13 dsl_scan_scrub_cb (dp=<optimized out>, bp=0xfffffe0256bd9000, zb=<unavailable>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:3992
#14 0xffffffff82217f3c in dsl_scan_visitbp (bp=bp@entry=0xfffffe0256bd9000, zb=zb@entry=0xfffffe024d4db250, dnp=dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000,
    scn=scn@entry=0xfffff80215809c00, ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2042
#15 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffffe025792ec00, bp=0xfffff802a0214e00, zb=0xfffffe024d4db300,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#16 dsl_scan_visitbp (bp=bp@entry=0xfffffe025792ec40, zb=zb@entry=0xfffffe024d4db300, dnp=dnp@entry=0xfffffe025792ec00, ds=ds@entry=0xfffff801f7809000,
    scn=<unavailable>, scn@entry=0xfffff80215809c00, ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#17 0xffffffff822180f3 in dsl_scan_visitdnode (scn=0xfffff80215809c00, ds=0xfffff801f7809000, dnp=0xfffffe025792ec00, object=952990, ostype=<optimized out>,
    tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1951
--Type <RET> for more, q to quit, c to continue without paging--
#18 dsl_scan_recurse (scn=0xfffff80215809c00, ds=0xfffff801f7809000, dnp=<optimized out>, bp=0xfffff802918b5100, zb=0xfffffe024d4db3b0, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1893
#19 dsl_scan_visitbp (bp=bp@entry=0xfffffe0256f1da00, zb=zb@entry=0xfffffe024d4db3b0, dnp=dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000, scn=scn@entry=0xfffff80215809c00, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#20 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffff8029fe68000, bp=0xfffff802887f0100, zb=0xfffffe024d4db460, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#21 dsl_scan_visitbp (bp=bp@entry=0xfffffe0257a5fe80, zb=zb@entry=0xfffffe024d4db460, dnp=0xfffff8029fe68000, dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000, scn=<unavailable>, scn@entry=0xfffff80215809c00,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#22 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffff8029fe68000, bp=0xfffff80291964180, zb=0xfffffe024d4db510, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#23 dsl_scan_visitbp (bp=bp@entry=0xfffffe02574ff000, zb=zb@entry=0xfffffe024d4db510, dnp=0xfffff8029fe68000, dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000, scn=<unavailable>, scn@entry=0xfffff80215809c00,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#24 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffff8029fe68000, bp=0xfffff802a63a8d00, zb=0xfffffe024d4db5c0, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#25 dsl_scan_visitbp (bp=bp@entry=0xfffffe02542a0000, zb=zb@entry=0xfffffe024d4db5c0, dnp=0xfffff8029fe68000, dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000, scn=<unavailable>, scn@entry=0xfffff80215809c00,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#26 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffff8029fe68000, bp=0xfffff80292b37480, zb=0xfffffe024d4db670, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#27 dsl_scan_visitbp (bp=bp@entry=0xfffffe02573cf000, zb=zb@entry=0xfffffe024d4db670, dnp=0xfffff8029fe68000, dnp@entry=0xfffff801f7809000, ds=0xfffff801f7809000, scn=<unavailable>, scn@entry=0xfffff80215809c00,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#28 0xffffffff82217e96 in dsl_scan_recurse (scn=<unavailable>, ds=0xfffff801f7809000, dnp=0xfffff8029fe68000, bp=0xfffff8029fc03380, zb=0xfffffe024d4db720, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1868
#29 dsl_scan_visitbp (bp=bp@entry=0xfffff8029fe68040, zb=zb@entry=0xfffffe024d4db720, dnp=dnp@entry=0xfffff8029fe68000, ds=ds@entry=0xfffff801f7809000, scn=<unavailable>, scn@entry=0xfffff80215809c00,
    ostype=<optimized out>, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
#30 0xffffffff82217dae in dsl_scan_visitdnode (scn=0xfffff80215809c00, ds=0xfffff801f7809000, dnp=<unavailable>, object=0, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1951
#31 dsl_scan_recurse (scn=0xfffff80215809c00, ds=0xfffff801f7809000, dnp=<optimized out>, bp=0xfffff802a7117400, zb=0xfffffe024d4db790, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:1912
#32 dsl_scan_visitbp (bp=bp@entry=0xfffff80108225e00, zb=zb@entry=0xfffffe024d4db790, dnp=dnp@entry=0x0, ds=ds@entry=0xfffff801f7809000, scn=scn@entry=0xfffff80215809c00, ostype=<optimized out>, tx=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2016
--Type <RET> for more, q to quit, c to continue without paging--
#33 0xffffffff822170af in dsl_scan_visit_rootbp (scn=scn@entry=0xfffff80215809c00, ds=0xfffff801f7809000, bp=0xfffff80108225e00, tx=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2071
#34 0xffffffff82217320 in dsl_scan_visitds (scn=scn@entry=0xfffff80215809c00, dsobj=11, tx=tx@entry=0xfffff8023711ee00) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2454
#35 0xffffffff82214286 in dsl_scan_visit (scn=scn@entry=0xfffff80215809c00, tx=<unavailable>, tx@entry=0xfffff8023711ee00) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:2737
#36 0xffffffff8221339b in dsl_scan_sync (dp=dp@entry=0xfffff80215ec5000, tx=tx@entry=0xfffff8023711ee00) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_scan.c:3717
#37 0xffffffff822411c9 in spa_sync_iterate_to_convergence (spa=0xfffffe024c90e000, tx=0xfffff8023711ee00) at /usr/src/sys/contrib/openzfs/module/zfs/spa.c:9091
#38 spa_sync (spa=spa@entry=0xfffffe024c90e000, txg=txg@entry=18748942) at /usr/src/sys/contrib/openzfs/module/zfs/spa.c:9287
#39 0xffffffff82255aa1 in txg_sync_thread (arg=arg@entry=0xfffff80215ec5000) at /usr/src/sys/contrib/openzfs/module/zfs/txg.c:591
#40 0xffffffff80bc2fce in fork_exit (callout=0xffffffff82255820 <txg_sync_thread>, arg=0xfffff80215ec5000, frame=0xfffffe024d4dbf40) at /usr/src/sys/kern/kern_fork.c:1093
#41 <signal handler called>
#42 0x60b1be6e7ced0484 in ?? ()
Backtrace stopped: Cannot access memory at address 0x86369ce92113d694
(kgdb)

Probably related: #13483
@nbari nbari added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jun 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nbari