NetBSD kernel coredumps

Working witth NetBSD coredumps

Here is complete step-by-step guide how to make kernel coredumps on NetBSD and how to get full stackraces with source lines in GDB.

Requirements:

• Host: KVM/LibVirt - used for serial port emulation (only device that can survive most panics) and also QEMU is able to send BREAK signal to virtual serial port to invoke NetBSD DDB - required when NetBSD kernel just deadlocks (but don't panic)
• Guest: any NetBSD of choice - testing CURRENT and 11.0_RC6

WARNING! Any kind of NetBSD disruption (panic or forced kernel core dumps) may corrupt filesystem to such extend that it will no longer boot! Even when you use log mode (WAPBL)! Always backup all your data before you plan to crash or coredump NetBSD kernel!

Preparation

You should do following to prepare NetBSD guest for usable coredumps:

1. Enable DDB on panic. Ensure that /etc/sysctl.conf contains:

   ddb.onpanic?=1
   

   After change you should either reboot or manually set this value:

   nbsd-head5$ doas sysctl -w ddb.onpanic=1
   
   ddb.onpanic: 0 -> 1

2. Next you need proper and kernel matching sources under /usr/src/sys. They are normally available in archive file syssrc.tgz

3. You need GDB version of kernel image to be able to open it in GDB and map all symbols from coredump. When you build your own kernel it is available in:

   /usr/src/sys/arch/amd64/compile/KERNEL_NAME/netbsd.gdb
   

   You will find that it is simply huge:

   $ ls -lhs 
   
   188M -rwxr-xr-x  1 ansible  users  188M Jul 25 06:38 netbsd.gdb

4. init file ~/.gdbinit (both in your user HOME and root's /root) I use following content:

set logging file ~/gdb.log set logging enabled on set pagination off set trace-commands on

Here is example how I build custom kernel:
```shell
# this will allow to do all builds as non-root user:
echo $USER

ansible

doas chown -R $USER /usr/src/sys/arch/amd64/conf /usr/src/sys/arch/amd64/compile
cd /usr/src/sys/arch/amd64/conf

# note I use HP name (my initials) for my kernel configuration:
cp GENERIC HP
vi HP # U uncomment all DEBUG options there: DEBUG, LOCKDEBUG, DIAGNOSTIC
config HP
cd ../compile/HP
make depend && make -j 1C

# finally gzip important files and copy kernel to root
gzip -9kv netbsd.gdb netbsd netbsd-HP.debug netbsd.map
# we will boot that kernel later:
# replace "hp1" with kernel name of your choice:
doas cp netbsd.gz /hp1

Booting kernel to be crashed

1. Shutdown NetBSD VM safely (doas halt -p)
2. Create Snapshot on Host (or full backup) of VM - important!!! As I already said - even when using WAPBL log option the FFS filesystem - it sometimes does not survive crash and is irrecoverable or even crashes kernel with panic! I already encountered that!

On Host I start logging session (I so I have logged all messages and stacktrace for future analysis):

Note: my NetBSD VM is called nbsd-head5 (HEAD snapshot 20260724074047Z).

# Host
script nbsd-head5-try3.log

virsh snapshot-create-as  nbsd-head5 safe-hp1try1
virsh start nbsd-head5

# Now switch quickly to virt-manager console and press '3' to drop to NetBSD
# boot prompt

# back on this serial port session start serial console:

virsh console nbsd-head5
Connected to domain 'nbsd-head5'
Escape character is ^] (Ctrl + ])

# now back on virt-mangaer console in Boot prompt switch to serial port:
> consdev com0

# you should see NetBSD Boot loader prompt (>) on your serial port session
# now boot our custom kernel (hp1) that we have debug symbols:
> boot hp1

Now we will try to crash (or deadlock) NetBSD guest kernel. I will use my procedure for deadlock (cause is not yet known):

# serial console session
# be sure to note what kernel version you booted:
nbsd-head5# uname -v
NetBSD 11.99.7 (HP) #0: Fri Jul 24 18:26:10 UTC 2026  root@nbsd-head5.example.com:/usr/src/sys/arch/amd64/compile/HP

# now I run this on serial console:
fssconfig -cx fss0 / /root/backup
# [ 223.8872520] /: suspended 0.181 sec, redo 0 of 322
dd if=/dev/fss0 of=/dev/null bs=1024k

# while DD is running I will do that on virt-manger console:
login: root
# after you are logged in just "exit" it

Above steps were enough to deadlock kernel - local console still accept characters but there is no I/O and all processes are stuck. To force DDB prompt we have to send BREAK signal over serial port.

Back on Linux host we have to do this (my VM name is nbsd-head5):

$ virsh qemu-monitor-command nbsd-head5  --hmp "info chardev"

charchannel1: filename=spicevmc
charchannel0: filename=disconnected:unix:/run/libvirt/qemu/channel/2-nbsd-head5/org.qemu.guest_agent.0,server=on
charserial0: filename=pty:/dev/pts/12
charmonitor: filename=unix:/var/lib/libvirt/qemu/domain-2-nbsd-head5/monitor.sock,server=on

$ virsh qemu-monitor-command nbsd-head5 --hmp "chardev-send-break charserial0

Last above command should trigger DDB prompt on our serial console. Here we should run few commands to note important data:

bt
show all procs/L
show all locks/t
show sleepq
# crashes with ,0x25048b486590c3c9address 0x25048b486590c4b9 is invalid
show tstiles

Note: if you forgot some commands there is no need to panic - we can issue some of them via crash(8) later.

Now we will force crash-dump (saved to swap partition) and reboot

• remember that this step can irrecoverably corrupt filesystem
• we have to boot SAME kernel that crashed to properly save coredump to files

To force crash dump run in DDB:

sync

Now again:

• in virt-manager console, watch for NetBSD boot manager
• quickly press 3 there to drop to boot prompt
• in boot prompt set serial console with consdev com0
• back in serial console boot proper kernel (same kernel that we core dumped) in my case with: boot hp1

If you are unlucky like me, you will see kernel panic on fileystem:

/dev/rld0a: file system is journaled; not checking
[   3.8963534] /: replaying log to disk
[   4.3799978] ufs_inactive: unlinked ino 7102730 on "/" has non zero size 0 or blocks ffffffffffffffa0 with a
llerror 0
[   4.3911509] panic: ufs_inactive: dirty filesystem?
[   4.3911509] cpu0: Begin traceback...
[   4.3911509] vpanic() at netbsd:vpanic+0x171
[   4.3911509] panic() at netbsd:panic+0x3c
[   4.3911509] ufs_inactive() at netbsd:ufs_inactive+0x173
[   4.3911509] VOP_INACTIVE() at netbsd:VOP_INACTIVE+0x3c
[   4.4000705] vrelel() at netbsd:vrelel+0x1ae
[   4.4000705] ffs_wapbl_replay_finish() at netbsd:ffs_wapbl_replay_finish+0x49
[   4.4000705] ffs_wapbl_start() at netbsd:ffs_wapbl_start+0x4a
[   4.4000705] ffs_mount() at netbsd:ffs_mount+0x57f
[   4.4000705] do_sys_mount() at netbsd:do_sys_mount+0x53c
[   4.4000705] sys___mount50() at netbsd:sys___mount50+0x36
[   4.4102679] syscall() at netbsd:syscall+0x9d
[   4.4102679] --- syscall (number 410) ---

In such case you have to:

• Force Power Off of VM
• revert snapshot
• try to again generate Crash dump

After several tries you see on boot that there was detected core-dump and at will be saved to /var/crash

TODO: more...