NetBSD kernel coredumps

Working witth NetBSD coredumps

Here is complete step-by-step guide how to make kernel coredumps on NetBSD and how to get full stackraces with source lines in GDB.

Requirements:

• Host: KVM/LibVirt - used for serial port emulation (only device that can survive most panics) and also QEMU is able to send BREAK signal to virtual serial port to invoke NetBSD DDB - required when NetBSD kernel just deadlocks (but don't panic)
• Guest: any NetBSD of choice - testing CURRENT and 11.0_RC6

WARNING! Any kind of NetBSD disruption (panic or forced kernel core dumps) may corrupt filesystem to such extend that it will no longer boot! Even when you use log mode (WAPBL)! Always backup all your data before you plan to crash or coredump NetBSD kernel!

Preparation

You should do following to prepare NetBSD guest for usable coredumps:

1. Enable DDB on panic. Ensure that /etc/sysctl.conf contains:

   ddb.onpanic?=1
   

   After change you should either reboot or manually set this value:

   nbsd-head5$ doas sysctl -w ddb.onpanic=1
   
   ddb.onpanic: 0 -> 1

2. Next you need proper and kernel matching sources under /usr/src/sys. They are normally available in archive file syssrc.tgz

3. You need GDB version of kernel image to be able to open it in GDB and map all symbols from coredump. When you build your own kernel it is available in:

   /usr/src/sys/arch/amd64/compile/KERNEL_NAME/netbsd.gdb
   

   You will find that it is simply huge:

   $ ls -lhs 
   
   188M -rwxr-xr-x  1 ansible  users  188M Jul 25 06:38 netbsd.gdb

4. init file ~/.gdbinit (both in your user HOME and root's /root) I use following content:

set logging file ~/gdb.log set logging enabled on set pagination off set trace-commands on

Here is example how I build custom kernel:
```shell
# this will allow to do all builds as non-root user:
echo $USER

ansible

doas chown -R $USER /usr/src/sys/arch/amd64/conf /usr/src/sys/arch/amd64/compile
cd /usr/src/sys/arch/amd64/conf

# note I use HP name (my initials) for my kernel configuration:
cp GENERIC HP
vi HP # U uncomment all DEBUG options there: DEBUG, LOCKDEBUG, DIAGNOSTIC
config HP
cd ../compile/HP
make depend && make -j 1C

# finally gzip important files and copy kernel to root
gzip -9kv netbsd.gdb netbsd netbsd-HP.debug netbsd.map
# we will boot that kernel later:
# replace "hp1" with kernel name of your choice:
doas cp netbsd.gz /hp1

Booting kernel to be crashed

1. Shutdown NetBSD VM safely (doas halt -p)
2. Create Snapshot on Host (or full backup) of VM - important!!! As I already said - even when using WAPBL log option the FFS filesystem - it sometimes does not survive crash and is irrecoverable or even crashes kernel with panic! I already encountered that!

On Host I start logging session (I so I have logged all messages and stacktrace for future analysis):

Note: my NetBSD VM is called nbsd-head5 (HEAD snapshot 20260724074047Z).

# Host
script nbsd-head5-try3.log

virsh snapshot-create-as  nbsd-head5 safe-hp1try1
virsh start nbsd-head5

# Now switch quickly to virt-manager console and press '3' to drop to NetBSD
# boot prompt

# back on this serial port session start serial console:

virsh console nbsd-head5
Connected to domain 'nbsd-head5'
Escape character is ^] (Ctrl + ])

# now back on virt-mangaer console in Boot prompt switch to serial port:
> consdev com0

# you should see NetBSD Boot loader prompt (>) on your serial port session
# now boot our custom kernel (hp1) that we have debug symbols:
> boot hp1

Now we will try to crash (or deadlock) NetBSD guest kernel. I will use my procedure for deadlock (cause is not yet known):

# serial console session
# be sure to note what kernel version you booted:
nbsd-head5# uname -v
NetBSD 11.99.7 (HP) #0: Fri Jul 24 18:26:10 UTC 2026  root@nbsd-head5.example.com:/usr/src/sys/arch/amd64/compile/HP

# now I run this on serial console:
fssconfig -cx fss0 / /root/backup
# [ 223.8872520] /: suspended 0.181 sec, redo 0 of 322
dd if=/dev/fss0 of=/dev/null bs=1024k

# while DD is running I will do that on virt-manger console:
login: root
# after you are logged in just "exit" it

Above steps were enough to deadlock kernel - local console still accept characters but there is no I/O and all processes are stuck. To force DDB prompt we have to send BREAK signal over serial port.

Back on Linux host we have to do this (my VM name is nbsd-head5):

$ virsh qemu-monitor-command nbsd-head5  --hmp "info chardev"

charchannel1: filename=spicevmc
charchannel0: filename=disconnected:unix:/run/libvirt/qemu/channel/2-nbsd-head5/org.qemu.guest_agent.0,server=on
charserial0: filename=pty:/dev/pts/12
charmonitor: filename=unix:/var/lib/libvirt/qemu/domain-2-nbsd-head5/monitor.sock,server=on

$ virsh qemu-monitor-command nbsd-head5 --hmp "chardev-send-break charserial0

Last above command should trigger DDB prompt on our serial console. Here we should run few commands to note important data:

bt
show all procs/L
show all locks/t
show sleepq
# crashes with ,0x25048b486590c3c9address 0x25048b486590c4b9 is invalid
show tstiles

Note: if you forgot some commands there is no need to panic - we can issue some of them via crash(8) later.

Now we will force crash-dump (saved to swap partition) and reboot

• remember that this step can irrecoverably corrupt filesystem
• we have to boot SAME kernel that crashed to properly save coredump to files

To force crash dump run in DDB:

sync

Now again:

• in virt-manager console, watch for NetBSD boot manager
• quickly press 3 there to drop to boot prompt
• in boot prompt set serial console with consdev com0
• back in serial console boot proper kernel (same kernel that we core dumped) in my case with: boot hp1

If you are unlucky like me, you will see kernel panic on fileystem:

/dev/rld0a: file system is journaled; not checking
[   3.8963534] /: replaying log to disk
[   4.3799978] ufs_inactive: unlinked ino 7102730 on "/" has non zero size 0 or blocks ffffffffffffffa0 with a
llerror 0
[   4.3911509] panic: ufs_inactive: dirty filesystem?
[   4.3911509] cpu0: Begin traceback...
[   4.3911509] vpanic() at netbsd:vpanic+0x171
[   4.3911509] panic() at netbsd:panic+0x3c
[   4.3911509] ufs_inactive() at netbsd:ufs_inactive+0x173
[   4.3911509] VOP_INACTIVE() at netbsd:VOP_INACTIVE+0x3c
[   4.4000705] vrelel() at netbsd:vrelel+0x1ae
[   4.4000705] ffs_wapbl_replay_finish() at netbsd:ffs_wapbl_replay_finish+0x49
[   4.4000705] ffs_wapbl_start() at netbsd:ffs_wapbl_start+0x4a
[   4.4000705] ffs_mount() at netbsd:ffs_mount+0x57f
[   4.4000705] do_sys_mount() at netbsd:do_sys_mount+0x53c
[   4.4000705] sys___mount50() at netbsd:sys___mount50+0x36
[   4.4102679] syscall() at netbsd:syscall+0x9d
[   4.4102679] --- syscall (number 410) ---

In such case you have to:

• Force Power Off of VM
• add new virtual disk to VM - I used size 4GB - we will copy dump there - we need new disk so it will survive snapshot-revert
  • in virt-manager: Add Hardware -> Storage -> Size: 4GB, type Disk, Bus type VirtIO
• boot from NetBSD ISO (you can use same consdev com0 trick to log all commands on serial console, followed by just boot command...
• when asked for terminal I used xterm
• confirm English message
• go to Utility Menu
• select Run /bin/sh

Now we sill partition 2nd disk and store whole swap (where is awaiting crash dump) to it:

# sysctl hw.disknames

hw.disknames = ld0 ld1 cd0

( our is (ld1) )
( find size )

# fdisk ld1 | grep 'total sectors'

fdisk: primary partition table invalid, no magic in sector 0
fdisk: Cannot determine the number of heads
total sectors: 8388608, bytes/sector: 512
total sectors: 8388608

( minus 64 from size for MBR and disklabel) :

# echo $(( 8388608 - 64 ))

8388544

( create MBR partition with boot block (not required) )

# fdisk -f -u -0 -i -s 169/64/8388544 -c /usr/mdec/mbr -b 1023/255/63  ld1

# fdisk ld1

Disk: /dev/rld1
NetBSD disklabel disk geometry:
cylinders: 8322, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 8388608, bytes/sector: 512

BIOS disk geometry:
cylinders: 523, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 8388608

Partitions aligned to 2048 sector boundaries, offset 64

Partition table:
0: NetBSD (sysid 169)
    start 64, size 8388544 (4096 MB, Cyls 0-522/42/32)
        PBR is not bootable: All bytes are identical (0x00)
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>
Bootselector disabled.
No active partition.
Drive serial number: 0 (0x00000000)

(create disklabel)

# disklabel -i ld1

Enter '?' for help
partition>?
?       print this menu
A       adjust the label size to the max disk size
C       make partitions contiguous
E       print disk label and current partition table
I       change label information
L       list all known file system types
N       name the label
P       print current partition table
Q       quit
R       rounding (c)ylinders (s)ectors
W       write the current partition table
[a-p]   define named partition
partition>P
5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 c:   8388544        64     unused      0     0        # (Cyl.      0*-   8322*)
 d:   8388608         0     unused      0     0        # (Cyl.      0 -   8322*)
 e:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>a
Filesystem type [unused]: 4.4BSD
Invalid file system typename `4.4BSD'
Filesystem type [unused]: 4.2BSD
Start offset ('x' to start after partition 'x') [0c, 0s, 0M]: 64
Partition size ('$' for all remaining) [0c, 0s, 0M]: 8388544
 a:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>P
5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
 c:   8388544        64     unused      0     0        # (Cyl.      0*-   8322*)
 d:   8388608         0     unused      0     0        # (Cyl.      0 -   8322*)
 e:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>W
Label disk [n]?y
disklabel: partitions a and e overlap
Label written
partition>Q

(format new partition a)

# newfs -O 2 ld1a

(mount it)

# mount /dev/ld1a /mnt

(copy dump from swap partition)

# dd if=/dev/ld0b | gzip -9c > /mnt/crash-swap1.bin.gz
# umount /mnt

(safe shutdown)
# halt -p

Back on LibVirt Host:

• Remove our 2nd disk from VM (but do NOT delete contents!!!)
• revert snapshot back to usable filesystem, in my case: virsh snapshot-revert nbsd-head5 safe-hp1try1
• reattach our 2nd drive to VM: Add Hardware -> Storage -> Select or Create Custom Storage -> Browse -> Manage... -> select your correct pool (normal is default) and your image - in my case it is nbsd-head5-X.qcow2 with size of 4GB
• ensure that you have attached NetBSD ISO and enable it in Boot Options (also Enable Boot menu)
• now repeat steps to boot NetBSD from ISO and Run Shell
• now do that to copy crash coredump to our swap partition:

# sysctl hw.disknames
hw.disknames = ld0 ld1 cd0
# mount -r /dev/ld1a /mnt
# ls -lhs /mnt
total 43M
 43M -rw-r--r--  1 root  wheel   43M Jul 25 08:22 crash-swap1.bin.gz

(Now moment of truth - copy crash dump to our (reverted) NetBSD swap partition:)
# gzcat /mnt/crash-swap1.bin.gz | dd of=/dev/ld0b

(perform save shutdown):
# sync
# umount /mnt
# halt -p

• now back in LibVirt/KVM host - again remove that second drive (without removal of data) - so future snapshot revert will not affect it...
• finally we can boot system as usual (using matching kernel - hp1 in my case and watch if it will detect and save coredump

Checking for core dump...
Jul 25 08:39:01 nbsd-head5 savecore: reboot after panic: dump forced via kernel debugger
savecore: reboot after panic: dump forced via kernel debugger
savecore: system went down at Sat Jul 25 07:58:51 2026

savecore: /var/crash/bounds: No such file or directory
savecore: writing compressed core to /var/crash/netbsd.0.core.gz
savecore: writing compressed kernel to /var/crash/netbsd.0.gz