Skip to content

NetBSD kernel coredumps

Henryk Paluch edited this page Jul 25, 2026 · 5 revisions

Working witth NetBSD coredumps

Here is complete step-by-step guide how to make kernel coredumps on NetBSD and how to get full stackraces with source lines in GDB.

Requirements:

  • Host: KVM/LibVirt - used for serial port emulation (only device that can survive most panics) and also QEMU is able to send BREAK signal to virtual serial port to invoke NetBSD DDB - required when NetBSD kernel just deadlocks (but don't panic)
  • Guest: any NetBSD of choice - testing CURRENT and 11.0_RC6

WARNING! Any kind of NetBSD disruption (panic or forced kernel core dumps) may corrupt filesystem to such extend that it will no longer boot! Even when you use log mode (WAPBL)! Always backup all your data before you plan to crash or coredump NetBSD kernel!

Preparation

You should do following to prepare NetBSD guest for usable coredumps:

  1. Enable DDB on panic. Ensure that /etc/sysctl.conf contains:

    ddb.onpanic?=1
    

    After change you should either reboot or manually set this value:

    nbsd-head5$ doas sysctl -w ddb.onpanic=1
    
    ddb.onpanic: 0 -> 1
  2. Next you need proper and kernel matching sources under /usr/src/sys. They are normally available in archive file syssrc.tgz

  3. You need GDB version of kernel image to be able to open it in GDB and map all symbols from coredump. When you build your own kernel it is available in:

    /usr/src/sys/arch/amd64/compile/KERNEL_NAME/netbsd.gdb
    

    You will find that it is simply huge:

    $ ls -lhs 
    
    188M -rwxr-xr-x  1 ansible  users  188M Jul 25 06:38 netbsd.gdb
  4. init file ~/.gdbinit (both in your user HOME and root's /root) I use following content:

    set logging file ~/gdb.log
    set logging enabled on
    set pagination off
    set trace-commands on
    

Here is example how I build custom kernel:

# this will allow to do all builds as non-root user:
echo $USER

  ansible

doas chown -R $USER /usr/src/sys/arch/amd64/conf /usr/src/sys/arch/amd64/compile
cd /usr/src/sys/arch/amd64/conf

# note I use HP name (my initials) for my kernel configuration:
cp GENERIC HP
vi HP # I uncomment all DEBUG options there: DEBUG, LOCKDEBUG, DIAGNOSTIC
config HP
cd ../compile/HP
make depend && make -j 1C

# finally gzip important files and copy kernel to root
gzip -9kv netbsd.gdb netbsd netbsd-HP.debug netbsd.map
# we will boot that kernel later:
# replace "hp1" with kernel name of your choice:
doas cp netbsd.gz /hp1

Booting kernel to be crashed

  1. Shutdown NetBSD VM safely (doas halt -p)
  2. Create Snapshot on Host (or full backup) of VM - important!!! As I already said - even when using WAPBL log option the FFS filesystem - it sometimes does not survive crash and is irrecoverable or even crashes kernel with panic! I already encountered that!

On Host I start logging session (I so I have logged all messages and stacktrace for future analysis):

Note: my NetBSD VM is called nbsd-head5 (HEAD snapshot 20260724074047Z).

# Host
script nbsd-head5-try3.log

virsh snapshot-create-as  nbsd-head5 safe-hp1try1
virsh start nbsd-head5

# Now switch quickly to virt-manager console and press '3' to drop to NetBSD
# boot prompt

# back on this serial port session start serial console:

virsh console nbsd-head5
Connected to domain 'nbsd-head5'
Escape character is ^] (Ctrl + ])

# now back on virt-mangaer console in Boot prompt switch to serial port:
> consdev com0

# you should see NetBSD Boot loader prompt (>) on your serial port session
# now boot our custom kernel (hp1) that we have debug symbols:
> boot hp1

Now we will try to crash (or deadlock) NetBSD guest kernel. I will use my procedure for deadlock (cause is not yet known):

# serial console session
# be sure to note what kernel version you booted:
nbsd-head5# uname -v
NetBSD 11.99.7 (HP) #0: Fri Jul 24 18:26:10 UTC 2026  root@nbsd-head5.example.com:/usr/src/sys/arch/amd64/compile/HP

# now I run this on serial console:
fssconfig -cx fss0 / /root/backup
# [ 223.8872520] /: suspended 0.181 sec, redo 0 of 322
dd if=/dev/fss0 of=/dev/null bs=1024k

# while DD is running I will do that on virt-manger console:
login: root
# after you are logged in just "exit" it

Above steps were enough to deadlock kernel - local console still accept characters but there is no I/O and all processes are stuck. To force DDB prompt we have to send BREAK signal over serial port.

Back on Linux host we have to do this (my VM name is nbsd-head5):

$ virsh qemu-monitor-command nbsd-head5  --hmp "info chardev"

charchannel1: filename=spicevmc
charchannel0: filename=disconnected:unix:/run/libvirt/qemu/channel/2-nbsd-head5/org.qemu.guest_agent.0,server=on
charserial0: filename=pty:/dev/pts/12
charmonitor: filename=unix:/var/lib/libvirt/qemu/domain-2-nbsd-head5/monitor.sock,server=on

$ virsh qemu-monitor-command nbsd-head5 --hmp "chardev-send-break charserial0

Last above command should trigger DDB prompt on our serial console. Here we should run few commands to note important data:

bt
show all procs/L
show all locks/t
show sleepq
# crashes with ,0x25048b486590c3c9address 0x25048b486590c4b9 is invalid
show tstiles

Note: if you forgot some commands there is no need to panic - we can issue some of them via crash(8) later.

Now we will force crash-dump (saved to swap partition) and reboot

  • remember that this step can irrecoverably corrupt filesystem
  • we have to boot SAME kernel that crashed to properly save coredump to files

To force crash dump run in DDB:

sync

Now again:

  • in virt-manager console, watch for NetBSD boot manager
  • quickly press 3 there to drop to boot prompt
  • in boot prompt set serial console with consdev com0
  • back in serial console boot proper kernel (same kernel that we core dumped) in my case with: boot hp1

If you are unlucky like me, you will see kernel panic on fileystem:

/dev/rld0a: file system is journaled; not checking
[   3.8963534] /: replaying log to disk
[   4.3799978] ufs_inactive: unlinked ino 7102730 on "/" has non zero size 0 or blocks ffffffffffffffa0 with a
llerror 0
[   4.3911509] panic: ufs_inactive: dirty filesystem?
[   4.3911509] cpu0: Begin traceback...
[   4.3911509] vpanic() at netbsd:vpanic+0x171
[   4.3911509] panic() at netbsd:panic+0x3c
[   4.3911509] ufs_inactive() at netbsd:ufs_inactive+0x173
[   4.3911509] VOP_INACTIVE() at netbsd:VOP_INACTIVE+0x3c
[   4.4000705] vrelel() at netbsd:vrelel+0x1ae
[   4.4000705] ffs_wapbl_replay_finish() at netbsd:ffs_wapbl_replay_finish+0x49
[   4.4000705] ffs_wapbl_start() at netbsd:ffs_wapbl_start+0x4a
[   4.4000705] ffs_mount() at netbsd:ffs_mount+0x57f
[   4.4000705] do_sys_mount() at netbsd:do_sys_mount+0x53c
[   4.4000705] sys___mount50() at netbsd:sys___mount50+0x36
[   4.4102679] syscall() at netbsd:syscall+0x9d
[   4.4102679] --- syscall (number 410) ---

In such case you have to:

  • Force Power Off of VM
  • add new virtual disk to VM - I used size 4GB - we will copy dump there - we need new disk so it will survive snapshot-revert
    • in virt-manager: Add Hardware -> Storage -> Size: 4GB, type Disk, Bus type VirtIO
  • boot from NetBSD ISO (you can use same consdev com0 trick to log all commands on serial console, followed by just boot command...
  • when asked for terminal I used xterm
  • confirm English language
  • go to Utility Menu
  • select Run /bin/sh

Now we will partition 2nd disk and store whole swap (where is awaiting crash dump) to it:

# sysctl hw.disknames

hw.disknames = ld0 ld1 cd0

( our is (ld1) )
( find size )

# fdisk ld1 | grep 'total sectors'

fdisk: primary partition table invalid, no magic in sector 0
fdisk: Cannot determine the number of heads
total sectors: 8388608, bytes/sector: 512
total sectors: 8388608

( minus 64 from size for MBR and disklabel) :

# echo $(( 8388608 - 64 ))

8388544

( create MBR partition with boot block (not required) )

# fdisk -f -u -0 -i -s 169/64/8388544 -c /usr/mdec/mbr -b 1023/255/63  ld1

# fdisk ld1

Disk: /dev/rld1
NetBSD disklabel disk geometry:
cylinders: 8322, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 8388608, bytes/sector: 512

BIOS disk geometry:
cylinders: 523, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 8388608

Partitions aligned to 2048 sector boundaries, offset 64

Partition table:
0: NetBSD (sysid 169)
    start 64, size 8388544 (4096 MB, Cyls 0-522/42/32)
        PBR is not bootable: All bytes are identical (0x00)
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>
Bootselector disabled.
No active partition.
Drive serial number: 0 (0x00000000)

(create disklabel)

# disklabel -i ld1

Enter '?' for help
partition>?
?       print this menu
A       adjust the label size to the max disk size
C       make partitions contiguous
E       print disk label and current partition table
I       change label information
L       list all known file system types
N       name the label
P       print current partition table
Q       quit
R       rounding (c)ylinders (s)ectors
W       write the current partition table
[a-p]   define named partition
partition>P
5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 c:   8388544        64     unused      0     0        # (Cyl.      0*-   8322*)
 d:   8388608         0     unused      0     0        # (Cyl.      0 -   8322*)
 e:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>a
Filesystem type [unused]: 4.2BSD
Start offset ('x' to start after partition 'x') [0c, 0s, 0M]: 64
Partition size ('$' for all remaining) [0c, 0s, 0M]: 8388544
 a:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>P
5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
 c:   8388544        64     unused      0     0        # (Cyl.      0*-   8322*)
 d:   8388608         0     unused      0     0        # (Cyl.      0 -   8322*)
 e:   8388544        64     4.2BSD      0     0     0  # (Cyl.      0*-   8322*)
partition>W
Label disk [n]?y
disklabel: partitions a and e overlap
Label written
partition>Q

(format new partition a)

# newfs -O 2 ld1a

(mount it)

# mount /dev/ld1a /mnt

(copy dump from swap partition)

# dd if=/dev/ld0b | gzip -9c > /mnt/crash-swap1.bin.gz
# umount /mnt

(safe shutdown)
# halt -p

Back on LibVirt Host:

  • Remove our 2nd disk from VM (but do NOT delete contents!!!)
  • revert snapshot back to usable filesystem, in my case: virsh snapshot-revert nbsd-head5 safe-hp1try1
  • reattach our 2nd drive to VM: Add Hardware -> Storage -> Select or Create Custom Storage -> Browse -> Manage... -> select your correct pool (normal is default) and your image - in my case it is nbsd-head5-X.qcow2 with size of 4GB
  • ensure that you have attached NetBSD ISO and enable it in Boot Options (also Enable Boot menu)
  • now repeat steps to boot NetBSD from ISO and Run Shell
  • now do that to copy crash coredump to our swap partition:
# sysctl hw.disknames
hw.disknames = ld0 ld1 cd0
# mount -r /dev/ld1a /mnt
# ls -lhs /mnt
total 43M
 43M -rw-r--r--  1 root  wheel   43M Jul 25 08:22 crash-swap1.bin.gz

(Now moment of truth - copy crash dump to our (reverted) NetBSD swap partition:)
# gzcat /mnt/crash-swap1.bin.gz | dd of=/dev/ld0b

(perform save shutdown):
# sync
# umount /mnt
# halt -p
  • now back in LibVirt/KVM host - again remove that second drive (without removal of data) - so future snapshot revert will not affect it...
  • finally we can boot system as usual (using matching kernel - hp1 in my case and watch if it will detect and save coredump
Checking for core dump...
Jul 25 08:39:01 nbsd-head5 savecore: reboot after panic: dump forced via kernel debugger
savecore: reboot after panic: dump forced via kernel debugger
savecore: system went down at Sat Jul 25 07:58:51 2026

savecore: /var/crash/bounds: No such file or directory
savecore: writing compressed core to /var/crash/netbsd.0.core.gz
savecore: writing compressed kernel to /var/crash/netbsd.0.gz

Now we should save all important files before we lost them. We need these binaries:

nbsd-head5$ mkdir ~/crashes/nbsd-head5-try5
nbsd-head5$ cd ~/crashes/nbsd-head5-try5
nbsd-head5$ doas mv -v /var/crash/*.gz .

/var/crash/netbsd.0.core.gz -> ./netbsd.0.core.gz
/var/crash/netbsd.0.gz -> ./netbsd.0.gz

nbsd-head5$ cp -v /usr/src/sys/arch/amd64/compile/HP/netbsd.gdb.gz .

/usr/src/sys/arch/amd64/compile/HP/netbsd.gdb.gz -> ./netbsd.gdb.gz

nbsd-head5$ doas chmod +r *
nbsd-head5$ ls -lhs
 50M -rw-r--r--  1 root     wheel   50M Jul 25 08:39 netbsd.0.core.gz
896K -rw-r--r--  1 root     wheel  877K Jul 25 08:39 netbsd.0.gz
 76M -rwxr-xr-x  1 ansible  users   75M Jul 25 08:41 netbsd.gdb.gz
nbsd-head5$ 

You also need matching kernel sources under /usr/src/sys

Now uncompress all these files (because most tools don't understand gzipped files) - while keeping (-k) existing compressed files:

nbsd-head5$ gzip -dkv *.gz

We need to first find Process addresses (STRUCT LWP *) to be able to switch GDB to process (PID) of our interest. We can do that with this command:

Crash version 11.99.7, image version 11.99.7.
System panicked: dump forced via kernel debugger
Backtrace from time of crash is available.

PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
2175   2175 3   0         0   ffff8400b349ec00                 dd biolock
2229   2229 3   1       180   ffff8400b6144400                 sh wait
2270   2270 3   0       180   ffff8400b349e800                 sh wait
1698   1698 3   2       180   ffff8400b349e000              getty ttyraw
1599   1599 3   0       180   ffff8400b6a6e800              getty ttyraw
2110   2110 3   1       180   ffff8400b65f0c00              login fstcnt
2174   2174 3   1       180   ffff8400b65f0800              login wait
...
0         0 3   3       200   ffffffff81a8c180            swapper uvm

Now finally run GDB (but in proper directory) and invoke these commands:

$ cd /usr/src/sys/arch/amd64/compile/HP
$ gdb ~/crashes/nbsd-head5-try5/netbsd.gdb

(now real magic happens)
(gdb) target kvm ~/crashes/nbsd-head5-try5/netbsd.0.core

But you will see that we are in not interested in default PID - one that rebooted system. Here is poorly documented switch - we want to see this process:

PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
2175   2175 3   0         0   ffff8400b349ec00                 dd biolock

So in GDB run this command we have to specify STRUCT LWP * address prefixed with 0x as shown below:

(gdb) kvm proc 0xffff8400b349ec00
+kvm proc 0xffff8400b349ec00
0xffffffff80e37943 in mi_switch (l=l@entry=0xffff8400b349ec00) at ../../../../kern/kern_synch.c:811
811			prevlwp = cpu_switchto(l, newl, returning);
(gdb) bt
+bt
#0  0xffffffff80e37943 in mi_switch (l=l@entry=0xffff8400b349ec00) at ../../../../kern/kern_synch.c:811
#1  0xffffffff80e34279 in sleepq_block (timo=timo@entry=0, catch_p=catch_p@entry=false, syncobj=syncobj@entry=0xffffffff81592aa0 <cv_syncobj>, nlocks=nlocks@entry=0) at ../../../../kern/kern_sleepq.c:392
#2  0xffffffff80dec0e2 in cv_timedwait (cv=0xffff8400b4018550, mtx=0xffffffff81abf200 <bufcache_lock>, timo=0) at ../../../../kern/kern_condvar.c:226
#3  0xffffffff80ecbb87 in bbusy (bp=bp@entry=0xffff8400b40184d0, intr=intr@entry=false, timo=timo@entry=0, interlock=interlock@entry=0x0) at ../../../../kern/vfs_bio.c:2206
#4  0xffffffff80ecbcc5 in getblk (vp=vp@entry=0xffff8400b4b557c0, blkno=1405332, size=size@entry=2048, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) at ../../../../kern/vfs_bio.c:1219
#5  0xffffffff80ecc03b in bio_doread (vp=vp@entry=0xffff8400b4b557c0, blkno=<optimized out>, size=size@entry=2048, async=async@entry=0) at ../../../../kern/vfs_bio.c:692
#6  0xffffffff80ecc1dd in bread (vp=vp@entry=0xffff8400b4b557c0, blkno=<optimized out>, size=size@entry=2048, flags=flags@entry=0, bpp=bpp@entry=0xffffa20279634de8) at ../../../../kern/vfs_bio.c:754
#7  0xffffffff80f08f62 in spec_read (v=<optimized out>) at ../../../../miscfs/specfs/spec_vnops.c:1171
#8  0xffffffff80efa153 in VOP_READ (vp=vp@entry=0xffff8400b4b557c0, uio=uio@entry=0xffffa20279634f00, ioflag=ioflag@entry=0, cred=cred@entry=0xffff8400b7c3c7c0) at ../../../../kern/vnode_if.c:785
#9  0xffffffff80ef0a2a in vn_read (fp=<optimized out>, offset=0xffff8400b35c3680, uio=0xffffa20279634f00, cred=0xffff8400b7c3c7c0, flags=<optimized out>) at ../../../../kern/vfs_vnops.c:675
#10 0xffffffff80e8980b in dofileread (fd=fd@entry=3, fp=<optimized out>, buf=0x7cd0e0d83000, nbyte=1048576, offset=<optimized out>, flags=flags@entry=1, retval=retval@entry=0xffffa20279634fb0) at ../../../../kern/sys_generic.c:156
#11 0xffffffff80e898cd in sys_read (l=<optimized out>, uap=0xffffa20279635000, retval=0xffffa20279634fb0) at ../../../../kern/sys_generic.c:121
#12 0xffffffff805cd9bb in sy_call (sy=0xffffffff81a87be8 <sysent+72>, l=0xffff8400b349ec00, uap=0xffffa20279635000, rval=0xffffa20279634fb0) at ../../../../sys/syscallvar.h:65
#13 sy_invoke (sy=0xffffffff81a87be8 <sysent+72>, l=0xffff8400b349ec00, uap=0xffffa20279635000, rval=0xffffa20279634fb0, code=3) at ../../../../sys/syscallvar.h:94
#14 syscall (frame=0xffffa20279635000) at ../../../../arch/x86/x86/syscall.c:137
#15 0xffffffff8021025d in handle_syscall ()

What you now see - is detailed stacktrace of dd command.

Now we can do same for suspected process login that probably deadlocked on normal exit(2) (=sys_exit call):

# lookup process prom crash | ps:
PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
...
2110   2110 3   1       180   ffff8400b65f0c00              login fstcnt

# and enter in GDB:
(gdb) kvm proc 0xffff8400b65f0c00
+kvm proc 0xffff8400b65f0c00
0xffffffff80e37943 in mi_switch (l=l@entry=0xffff8400b65f0c00) at ../../../../kern/kern_synch.c:811
811			prevlwp = cpu_switchto(l, newl, returning);
(gdb) bt
+bt
#0  0xffffffff80e37943 in mi_switch (l=l@entry=0xffff8400b65f0c00) at ../../../../kern/kern_synch.c:811
#1  0xffffffff80e34279 in sleepq_block (timo=timo@entry=0, catch_p=catch_p@entry=true, syncobj=syncobj@entry=0xffffffff81592aa0 <cv_syncobj>, nlocks=nlocks@entry=0) at ../../../../kern/kern_sleepq.c:392
#2  0xffffffff80debef6 in cv_wait_sig (cv=cv@entry=0xffffffff81b9d540 <fstrans_count_cv>, mtx=mtx@entry=0xffffffff81abf480 <fstrans_lock>) at ../../../../kern/kern_condvar.c:203
#3  0xffffffff80ee84e3 in fstrans_setstate (mp=mp@entry=0xffff8400b65fc000, new_state=new_state@entry=FSTRANS_SUSPENDING) at ../../../../kern/vfs_trans.c:722
#4  0xffffffff80f05737 in genfs_suspendctl (mp=0xffff8400b65fc000, cmd=<optimized out>) at ../../../../miscfs/genfs/genfs_vfsops.c:82
#5  0xffffffff80ede894 in VFS_SUSPENDCTL (mp=0xffff8400b65fc000, a=a@entry=1) at ../../../../kern/vfs_subr.c:1666
#6  0xffffffff80ee8925 in vfs_suspend (mp=mp@entry=0xffff8400b65fc000, nowait=nowait@entry=0) at ../../../../kern/vfs_trans.c:784
#7  0xffffffff80eea258 in vrevoke_suspend_next (lastmp=<optimized out>, thismp=0xffff8400b65fc000) at ../../../../kern/vfs_vnode.c:1248
#8  0xffffffff80eecdf9 in vrevoke (vp=0xffff8400b5856840) at ../../../../kern/vfs_vnode.c:1272
#9  0xffffffff80f05c77 in genfs_revoke (v=<optimized out>) at ../../../../miscfs/genfs/genfs_vnops.c:310
#10 0xffffffff80efa6ff in VOP_REVOKE (vp=vp@entry=0xffff8400b5856840, flags=flags@entry=1) at ../../../../kern/vnode_if.c:1071
#11 0xffffffff80e042c7 in exit1 (l=l@entry=0xffff8400b65f0c00, exitcode=<optimized out>, signo=signo@entry=0) at ../../../../kern/kern_exit.c:420
#12 0xffffffff80e04766 in sys_exit (l=0xffff8400b65f0c00, uap=0xffffa202791e9000, retval=<optimized out>) at ../../../../kern/kern_exit.c:188
#13 0xffffffff805cd9bb in sy_call (sy=0xffffffff81a87bb8 <sysent+24>, l=0xffff8400b65f0c00, uap=0xffffa202791e9000, rval=0xffffa202791e8fb0) at ../../../../sys/syscallvar.h:65
#14 sy_invoke (sy=0xffffffff81a87bb8 <sysent+24>, l=0xffff8400b65f0c00, uap=0xffffa202791e9000, rval=0xffffa202791e8fb0, code=1) at ../../../../sys/syscallvar.h:94
#15 syscall (frame=0xffffa202791e9000) at ../../../../arch/x86/x86/syscall.c:137
#16 0xffffffff8021025d in handle_syscall ()

To get more details we can select specific frame and even list related source code!:

gdb) frame 11
+frame 11
#11 0xffffffff80e042c7 in exit1 (l=l@entry=0xffff8400b65f0c00, exitcode=<optimized out>, signo=signo@entry=0) at ../../../../kern/kern_exit.c:420
420					VOP_REVOKE(vprevoke, REVOKEALL);
(gdb) l
+l
415	
416			if (vprevoke != NULL || vprele != NULL) {
417				if (vprevoke != NULL) {
418					/* Releases proc_lock. */
419					proc_sessrele(sp);
420					VOP_REVOKE(vprevoke, REVOKEALL);
421				} else
422					mutex_exit(&proc_lock);
423				if (vprele != NULL)
424					vrele(vprele);

(gdb) info locals
+info locals
vprele = 0xffff8400b5856840
vprevoke = 0xffff8400b5856840
sp = 0xffff8400b6378c80
tp = <optimized out>
p = 0xffff8400b6933740
child = <optimized out>
next_child = <optimized out>
old_parent = <optimized out>
new_parent = <optimized out>
pgrp = 0xffff8400b65ccfc0
ksi = {ksi_flags = 18446744071584691584, ksi_list = {tqe_next = 0xffffffff81593580 <__func__.4>, tqe_prev = 0xffffffff80e65c87 <ld_rbto_compare_key>}, ksi_info = {_signo = -2130408559, _code = -1, _errno = 0, _pad = 0, _reason = {_rt = {_pid = -1289693824, _uid = 4294935552, _value = {sival_int = 1750081536, sival_ptr = 0xffffa20268502000}}, _child = {_pid = -1289693824, _uid = 4294935552, _status = 1750081536, _utime = 4294943234, _stime = 2162181966}, _fault = {_addr = 0xffff8400b320d580, _trap = 1750081536, _trap2 = -24062, _trap3 = -2132785330}, _poll = {_band = -136336436570752, _fd = 1750081536}, _syscall = {_sysnum = -1289693824, _retval = {-31744, 1750081536}, _error = -24062, _args = {18446744071576766286, 18446607737327389696, 0, 18446640730238521184, 18446744071577168525, 707, 18446744071584691584, 0}}, _ptrace_state = {_pe_report_event = -1289693824, _option = {_pe_other_pid = -31744, _pe_lwp = -31744}}}}, ksi_lid = 1750081992}
kq = {tqh_first = 0x0, tqh_last = 0xffffa202791e8ea0}
wakeinit = <optimized out>
__func__ = "exit1"

Really good! And that's all...

Clone this wiki locally