Linux VM Kernel Debugging

These are some handy steps for doing Linux live kernel debugging on the OpenXT platform. OpenXT makes this readily doable by using serial over IP within QEMU to connect the remote debugger GDB to the kernel debugger KGDB. Of course everyone has their favorite distro but for brevity's sake we will stick to just one – Debian. Most of the steps should be the same with other flavors aside from the package management bits and specifics of rebuilding the kernel sources.

Beforehand

Here are some things to setup up first and to keep in mind while going through this:

• This is based on using Debian Wheezy HVMs. That means apt package manager and .deb package files.
• Throughout this guide:
• target is the VM that is being kernel debugged
• host is the debugger, the VM where GDB is remotely connecting to the target.
• For simplicity, it is assumed both the host and target are the same OS or distro. The paths and users on both are the same. Also password-less SSH login and sudo setup are assumed.
• A basic set of development tools is needed on the target to build the kernel - apt-get install build-essential should be sufficient (note sometimes dpkg-dev needs to be installed manually).

So to get started, install OpenXT and create 2 Debian Wheezy HVMs. For debugging to work, SELinux and stubdoms need to be turned off. Stubdoms are disabled in the Advanced tabs for the VMs. To disable SELinux, run a terminal, use nr to log into the admin role then run rw to make the rootfs read-write. Next edit /etc/selinux/config and set SELINUX=permissive and save. Reboot.

Building a Kernel

The KGDB debugger components need to be enabled in the target kernel. This requires building a custom kernel. On the target, get the kernel source package - in this case for Wheezy:

$ sudo apt-get install linux-source-3.2

This drops off a tarball /usr/srclinux-source-3.2.tar.bz2. Make a directory called ~/kernel and extract the tarball there. Change to ~/kernel/linux-source-3.2 that has the kernel sources.

Next the kernel sources need a configuration file. The simplest thing is to start with the one for the current installed Wheezy kernel. Copy /boot/config-3.2.0-4-amd64 as .config in the current sources dir. Note the current config file might have a different name. These are the setting that should be enabled/disabled in the ```.config````:

CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_DEBUG_RODATA is not set

Briefly, these are the base set of kernel debug features that need to be modified. CONFIG_DEBUG_RODATA marks the .text section RO and prevent KGDB from emitting break-point instruction in the code. CONFIG_DEBUG_INFO enabled debug information/symbolic data in the kernel image. CONFIG_FRAME_POINTER preserves stack frame pointers make stack back-tracing and changing frames much easier. The last two enable the KGDB debugger extensions using a serial console.

Though you can edit .config, it is usually done using one of the editing interfaces. Using make menuconfig, the setting are here:

• Kernel hacking
• Compile the kernel with debug info
• Compile the kernel with frame pointers
• KGDB: kernel debugger --->
• KGDB: use kgdb over the serial console
• Write protect kernel read-only data structures

Other debugging features may be enabled/disabled at this point too. Once all this is done, save the configuration and:

$ make deb-pkg

This will produce 3 packages in the ~/kernel dir:

• linux-image-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb
• linux-headers-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb
• linux-libc-dev_3.2.65-kgdb-1_amd64.deb

The first two are needed (the kernel and the headers needed to develop with this kernel).