kernel_debug

== kernel debug ==

Dealing with panics.

Boot target VM with "-v keepsyms=y debug=0x144"

Make it panic.

On your dev machine, you will need the kernelDebugKit from apple. Download it.

gdb /Volumes/Kernelit/mach_kernel
source /Volumes/KernelDebugKit/kgmacros
target remote-kdp

kdp-reattach  192.168.30.133   # obviously use the IP of your target / crashed VM

showallkmods
(find "address" for zfs and spl modules)
^Z   # suspend gdb, or, use another terminal

kextutil -s /tmp -n -k /Volumes/KernelDebugKit/mach_kernel -e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ ../spl/module/spl/spl.kext/

fg # resume gdb, or go back to gdb terminal
set kext-symbol-file-path /tmp

add-kext /tmp/spl.kext
add-kext /tmp/zfs.kext

bt

=== Non-panic ===

You can always panic a kernel with

sudo dtrace -w -n "BEGIN{ panic();}"

If you prefer to work in gdb.

But this was revealing;

#  /usr/libexec/stackshot -i -f /tmp/stackshot.log 
# symstacks.rb -f /tmp/stackshot.log -s -w /tmp/trace.txt
# less /tmp/trace.txt

Note that my hang is here;

PID: 156
    Process: zpool
    Thread ID: 0x4e2
    Thread state: 0x9 == TH_WAIT |TH_UNINT 
    Thread wait_event: 0xffffff8006608a6c
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 366 (0xffffff80002b3d3e)
      0xffffff800022e711 (in mach_kernel) + 1281 (0xffffff800022e711)
        thread_block_reason (in mach_kernel) + 300 (0xffffff800022d9dc)
          lck_mtx_sleep (in mach_kernel) + 78 (0xffffff80002265ce)
            0xffffff8000569ef6 (in mach_kernel) + 246 (0xffffff8000569ef6)
              msleep (in mach_kernel) + 116 (0xffffff800056a2e4)
                0xffffff7f80e52a76 (0xffffff7f80e52a76)
                  0xffffff7f80e53fae (0xffffff7f80e53fae)
                    0xffffff7f80e54173 (0xffffff7f80e54173)
                      0xffffff7f80f1a870 (0xffffff7f80f1a870)
                        0xffffff7f80f2bb4e (0xffffff7f80f2bb4e)
                          0xffffff7f80f1a9b7 (0xffffff7f80f1a9b7)
                            0xffffff7f80f1b65f (0xffffff7f80f1b65f)
                              0xffffff7f80f042ee (0xffffff7f80f042ee)
                                0xffffff7f80f45c5b (0xffffff7f80f45c5b)
                                  0xffffff7f80f4ce92 (0xffffff7f80f4ce92)
                                    spec_ioctl (in mach_kernel) + 157 (0xffffff8000320bfd)
                                      VNOP_IOCTL (in mach_kernel) + 244 (0xffffff8000311e84)

It is a little shame that it only shows the kernel symbols, and not inside SPL and ZFS. But we can ask it to load another sym file (alas, it can not handle multiple symbols files, fix this Apple)

# kextstat
# Grab the addresses of spl and zfs again
# kextutil -s /tmp -n -k /Volumes/KernelDebugKit/mach_kernel -e -r /Volumes/KernelDebugKit module/zfs/zfs.kext/ ../spl/module/spl/spl.kext/ 

# symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.spl.sym
              0xffffff800056a2e4 (0xffffff800056a2e4)
                spl_cv_wait (in net.lundman.spl.sym) + 54 (0xffffff7f80e52a76)
                  taskq_wait (in net.lundman.spl.sym) + 78 (0xffffff7f80e53fae)
                    taskq_destroy (in net.lundman.spl.sym) + 35 (0xffffff7f80e54173)
                      0xffffff7f80f1a870 (0xffffff7f80f1a870)

# symstacks.rb -f /tmp/stackshot.log -s -k /tmp/net.lundman.zfs.sym
                    0xffffff7f80e54173 (0xffffff7f80e54173)
                      vdev_open_children (in net.lundman.zfs.sym) + 336 (0xffffff7f80f1a870)
                        vdev_root_open (in net.lundman.zfs.sym) + 94 (0xffffff7f80f2bb4e)
                          vdev_open (in net.lundman.zfs.sym) + 311 (0xffffff7f80f1a9b7)
                            vdev_create (in net.lundman.zfs.sym) + 31 (0xffffff7f80f1b65f)
                              spa_create (in net.lundman.zfs.sym) + 878 (0xffffff7f80f042ee)

Voila!