-
Notifications
You must be signed in to change notification settings - Fork 42
Debugging the Linux kernel
The tips on this page focus on helping debugging Linux XIA kernel, but most content is general enough to help other projects. We've only tested this tips on Ubuntu but one should be able to adapt them for other platforms as well.
While developing, testing, and debugging a Linux kernel, one should run it in a VM. This will protect your host machine against kernel crashes and will enable easier and faster debugging. We recommend VirtualBox to deal with virtualization given that it has a friendly user interface, but one is free to pick another tool.
Using public and private keys in SSH for authentication is often recommended for security, but here we recommend it simply to save you time logging in your debugging VM.
In case you don't already have a set of keys, run the following commands in your home folder on the host:
$ mkdir ~/.ssh $ chmod 700 ~/.ssh $ ssh-keygen -t rsa
The first two commands may not be necessary in case folder .ssh already exists. Once you're prompted to enter a passphase, leave it blank; otherwise you'll deceive the whole purpose of this section.
Once the key set is available, just transfer the public key to the VM running the command below:
$ ssh-copy-id USERNAME@VM-ADDRESS
Notice that the "USERNAME@" part is optional if you use the same username on the host and VM. This should be the last time you enter anything for password while logging in your debugging VM. If you want to know more about this form of authentication, check out the page SSH/OpenSSH/Keys.
When Linux kernel crashes, it often writes valuable information to the console before it completely freezes; this information is known as crash dump, or oops message. Crash dumps are often bigger than what the console can show at once, and one can't scroll the screen to read because the kernel is dead. This section shows how to redirect the console to the first serial port so one can captured the dumps in the host.
The first step is to add a serial port to your VM, Under VirtualBox's VM settings,
- Select "Serial Ports"/"Port 1";
- Check "Enable Serial Port" and "Create Pipe";
- Field "Port Number" must be "COM1" and field "Port Mode" "Host Pipe";
- Finally, enter in "Port/File Path" the name of the pipe file in the host (e.g., /tmp/myvmpipe).
- If you haven't socat(1) already installed on your host machine, issue the following command: sudo apt-get install socat
In the VM, add the parameters "console=tty0 console=ttyS0,115200" to variable GRUB_CMDLINE_LINUX in file /etc/default/grub as shown below:
-GRUB_CMDLINE_LINUX="" +GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200"
Then run the command below, and reboot the VM to apply the change.
# update-grub
Done. Now, wherever you want to capture the console in the host, run "socat /tmp/myvmpipe -" and let it running in a terminal. Please notice that you must start capturing the console before a crash.
Once oops start to show up, you'd better be able to read them. The page Understanding a Kernel Oops! is a friendly entry point for that.
Disabling the console screen saver is useful for debugging because once the kernel crashes, there's no way of reading what's on the console if the screen saver was active during the crash. Need less to say that a console screen saver in a VM is pointless. If the console is already being redirected as explained above and the console redirection is always in use, disabling the screen saver is redundant.
To disable the console screen saver, add the parameter "consoleblank=0" to variable GRUB_CMDLINE_LINUX_DEFAULT in file /etc/default/grub in the VM as shown below:
-GRUB_CMDLINE_LINUX_DEFAULT="" +GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0"
Then run the command below, and reboot the VM to apply the change.
# update-grub
kgdb doesn't have as much documentation as its sister gdb(1). This gap can be partially avoided using gdb's documentation wherever it's possible. Nevertheless, for someone with little or no experience with gdb, its documentation is too dry to start; the book The Art of Debugging with GDB, DDD, and Eclipse can help one to get going, and prepared to dive into gdb's documentation when something specific surfaces. If you're not interested in DDD and Eclipse, save you time and skip those sections in the book. Quick introductions to gdb are also available on YouTube such as the followings:
- Introduction to GDB a tutorial - Harvard CS50;
- C Programming: Use gdb to debug C programs;
- GDB Debugger.
The steps below describe how to start a kgdb session in a host machine while the kernel being debugged is running in a virtual machine with VirtualBox. These steps are a refinement of the ones presented in the article KGDB with VirtualBox: Debug a Live Kernel. More information about running kgdb is found here.
One has to follow the steps below only once.
- Enable a second serial port in your VM; the first serial port is being used to redirect the console as explained in the section before. Under VirtualBox's VM settings,
- Select "Serial Ports"/"Port 2";
- Check "Enable Serial Port". Field "Port Number" must be "COM2";
- Set "Port Mode" to "Host Pipe", and check "Create Pipe";
- Finally, enter in "Port/File Path" the name of the pipe file in the host (e.g., /tmp/myvmpipe).
- If you haven't socat(1) already installed on your host machine, issue the following command: sudo apt-get install socat
If you have followed page How to install to prepare your XIA kernel, you're good to go; otherwise you should read section "Kernel config options for kgdb" of this page, and prepare your kernel accordingly.
Once a VM has booted with your kernel, but before you start a debugging session, run the following command in the VM as root:
# echo ttyS1,115200 > /sys/module/kgdboc/parameters/kgdboc
This will tell the kernel to use the second serial port for gdb. Whenever you want to start a debugging session, run the following command in the VM:
# echo g > /proc/sysrq-trigger
At this point you'll see kdb's prompt in the VM's console. You can learn more about kdb here, but this section is really about kgdb. Thus, enter kgdb in the console.
In the host, you need to call socat in the host to link the second serial port to a terminal.
socat -d -d /tmp/myvmpipe pty &
This command will run in background and output something like this:
$ socat -d -d ../xia1_serial2 pty & 2013/08/16 11:10:16 socat[11151] N opening connection to AF=1 "../xia1_serial2" 2013/08/16 11:10:16 socat[11151] N successfully connected from local address AF=1 "\xB7\x@8\x01" 2013/08/16 11:10:16 socat[11151] N successfully connected via 2013/08/16 11:10:16 socat[11151] N PTY is /dev/pts/6 2013/08/16 11:10:16 socat[11151] N starting data transfer loop with FDs [3,3] and [4,4]
The important information is "PTY is /dev/pts/6" that will be used next. In the folder in which the kernel's source is kept, run gdb passing "./vmlinux" as parameter. Once gdb prompt is available, enter "target remote /dev/pts/6" (change the PTY accordingly). Here's an example:
$ cd linux $ gdb ./vmlinux GNU gdb (GDB) 7.5-ubuntu Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/michel/git/xia/linux/vmlinux...done. (gdb) target remote /dev/pts/6
This connects gdb to the waiting kernel session in the VM.
If you need to debug a kernel module, very likely, insert the module in the VM, obtain the .text address of the module with the command "cat /sys/module/MODULE_NAME/sections/.text", and use it as an argument for the gdb command add-symbol-file. The example below does it for module xia:
In the VM:
# cat /sys/module/xia/sections/.text 0xffffffffa0133000
In the host:
(gdb) add-symbol-file net/xia/xia.o 0xffffffffa0133000
To help debugging XIA, you may want to use the following bash script in the VM:
#!/bin/bash # Enabling kgdb over the second serial port. echo ttyS1,115200 > /sys/module/kgdboc/parameters/kgdboc echo Debugger enabled at ttyS1. # Load modules. echo -e '\nmodprobe xia' modprobe xia for ppal in $@; do echo modprobe xia_ppal_$ppal modprobe xia_ppal_$ppal done # List commands to run from gdb. echo -e '\nadd-symbol-file net/xia/xia.o' `cat /sys/module/xia/sections/.text` for ppal in $@; do echo add-symbol-file net/xia/ppal_$ppal/xia_ppal_$ppal.o \ `cat /sys/module/xia_ppal_$ppal/sections/.text` done # Stop to synchronize with gdb. echo g > /proc/sysrq-trigger
To use this script, pass the names of the principals to be loaded as parameters. For example, assuming the name of the script is debug, the following command will load principals AD, HID, Serval and XDP:
./debug ad hid serval xdp
Linux's kernel cannot be compiled without compiling optimizations, and these optimizations do mess the debugging information. gdb handles this problem reasonably well, but one has to be fully aware of what's going on because gdb will show a behavior and values that will seem wrong. This section tries to accumulate information about this problem to help developers, but keep in mind that it's not exhaustive and your collaboration is very appreciated.
- When a function is static and only called once in a file, gcc will make this function inline. Thus, one cannot walk through this function because it's just a line for kgdb. Solution, drop the static qualifier for the functions that show this problem and you want to analyse while debugging.
Thanks to Linux XIA promoting principals to be mapped to kernel modules, principal developers can save time testing and debugging their principals. Instead of compiling the whole kernel for each change, developers can simply compile their principals and update the testing kernel without a reboot. Of course, this is only possible if the principal under test doesn't mess the running kernel up during the test and while the principal is being unloaded. This section presents this process through an example that can be easily adapted to each developer's need.
The first step is to compile and install the XIA kernel following the steps described in page How to install. This step is important to avoid binary mismatches between the (testing) running kernel and the compiled binaries. Notice that this step is also essential to use kgdb.
Once your local binary of the XIA kernel is the same of the running kernel, a principal developer can continue her work of writing a new principal, in our example here the Serval principal, and compile only the XIA code as follows:
$ make M=net/xia CC [M] net/xia/ppal_serval/serval_tcp_output.o LD [M] net/xia/ppal_serval/xia_ppal_serval.o Building modules, stage 2. MODPOST 6 modules CC net/xia/ppal_serval/xia_ppal_serval.mod.o LD [M] net/xia/ppal_serval/xia_ppal_serval.ko
Notice that is important to avoid using 'M=net/xia/ppal_serval' to give the Makefile script the chance to verify that the XIA module was not changed as well. There's a more technical aspect that we'll skip here that is that principals call functions exported by the xia module, and the entry point of these functions are kept in net/xia/Module.symvers. Finally, if the xia module was changed, the steps below must copy this module to account for this dependency.
On the testing VM as root, the old module can be unloaded as follows:
# rmmod xia_ppal_serval
If the xia module is also being updated, all principals must be unloaded before unloading the xia module.
The current practice in Linux XIA is for principals to notify when they are successfully loaded or unloaded via kernel log. Therefore, the step before can be verified with command dmesg as follows:
# dmesg | tail bla bla bla # lines omitted [ 8.122166] XIA Principal Serval UNloaded
A method developers can use to transfer a new module to the testing VM is to use the command scp as follows:
$ scp net/xia/ppal_serval/*.ko developer@192.168.1.8: xia_ppal_serval.ko 100% 3836KB 3.8MB/s 00:01
Once the new module is available on the VM, the following steps will move the new module to its proper place:
# chmod 644 xia_ppal_serval.ko # chown root:root xia_ppal_serval.ko # mv xia_ppal_serval.ko /lib/modules/`uname -r`/kernel/net/xia/ppal_serval/
The commands chmod and chown above are meant to avoid problem with security checks performed while loading principals into a kernel. Also notice that the `uname -r` takes care of which kernel has to be updated if more than one is installed on the VM; that is, we're updating the running kernel.
With everything in place, the new module can be loaded as follows:
# modprobe xia_ppal_serval
To verify that it was successfully loaded, one can use the command dmesg again as follows:
# dmesg | tail bla bla bla # lines omitted [ 8.122166] XIA Principal Serval UNloaded [ 10.818589] XIA Principal Serval loaded
Wireshark is a network protocol analyzer that lets users inspect packet data. It provides a GUI through which users can view conveniently-formatted packet information broken down by protocol.
It can be tedious to try to debug Linux XIA based on raw packet bytes. Instead, users can view XIP packets using the Wireshark application.
Currently, the upstream Wireshark application does not have XIA support. However, there is a personal package archive (PPA) that is available for download that does support XIA. The PPA currently supports the following Debian distributions and architectures:
- Ubuntu 12.10 Quantal Quetzal (amd64, i386)
- Ubuntu 13.04 Raring Ringtail (amd64, i386)
- Ubuntu 13.10 Saucy Salamander (amd64, i386)
- Ubuntu 14.04 Trusty Tahr (amd64, i386)
However, note that the later releases have greater functionality. For example, only the Ubuntu 14.04 version is capable of correctly dissecting XIP packets that contain Serval information. The older versions can still be useful, even if they are not technically up-to-date.
There are several options for obtaining Wireshark with XIA support:
You can download and install Wireshark with XIA using the Debian Advanced Packaging Tool (APT). To do this, you will have to add an entry into the source list of repositories; then, when you use a command such as apt-get to install Wireshark, you would then be downloading and installing the Wireshark-XIA version.
To do this, you will add an APT repository for the Wireshark-XIA PPA:
# add-apt-repository ppa:doucette-4/wireshark-xia
Once the repository and key have been added, you can run an update of APT to retrieve an updated list of all available software:
# apt-get update
Then, if you do not already have a copy of Wireshark, simply install Wireshark as you might normally:
# apt-get install wireshark
If you already have a copy of Wireshark, then see the section below about updating.
You can download and install a Wireshark application complete with the XIA packet dissectors in the Wireshark-XIA Launchpad repository.
You should first remove any official Wireshark application that you may have installed.
On the Wireshark-XIA PPA page, first click on "View Package Details." Then, find the series of the application that you would like, and click on the appropriate "wireshark-*" link under "Source" on the same page.
Under "Package Files" download both files ending in *_all.deb. Then, if your architecture is i386, download all files ending in *_i386.deb. Otherwise, download all files ending in *_amd64.
Once you have downloaded all of these files in the same directory, issue this command:
# dpkg -i *deb
Note: any packages used for documentation (doc), development (dev), or debugging (dbg) are not necessary, and can be safely excluded from downloading and the dpkg command..
If you are using the development version of Wireshark and want to add in the custom dissectors yourself, consult the wireshark-xia repository. The README file there will walk you through the steps.
There is plenty of documentation about how to use Wireshark available on the official website. The only difference between the official version of Wireshark and the XIA version is that the XIA version has added "packet dissectors" for the XIP and NWP protocols. Therefore, when a user opens a network capture that contains XIP and NWP packets, those packets will be displayed by the XIA version of Wireshark.
The XIP packet dissector will display all of the information in an XIP header. Some information about XIP address formats can be found here: Human readable XIP address format.
You may notice that some of the addresses displayed by Wireshark are marked as invalid by address syntax (with an '!' at the front). More information about this can be found in the FAQ.
If you already have Wireshark and wish to obtain Wireshark-XIA, execute the following commands:
# apt-get remove wireshark # apt-get remove wireshark-common # add-apt-repository ppa:doucette-4/wireshark-xia # apt-get update # apt-get install wireshark
If you want to revert your Wireshark-XIA application back to the official Wireshark branch, you need to remove the references to the Wireshark-XIA PPA from your sources list. You can then remove Wireshark and install again:
# add-apt-repository --remove ppa:doucette-4/wireshark-xia # apt-get remove wireshark # apt-get remove wireshark-common # apt-get update # apt-get install wireshark
You can also find a couple of sample XIP and NWP packet captures here.
- Chapter 18 of book Linux Kernel Development summarizes common debugging techniques used by Linux kernel developers.
- Ubuntu wiki's page Kernel Debugging Tricks includes useful kernel parameters for debugging, how to use USB as a serial port, how to find the location where your kernel panicked or oopsed, among other tricks.
- Page XIA Linux Containers (XLXC) shows how to simplify debugging of a couple machines using Linux Containers.
All grants that have generously supported the development of Linux XIA are listed on our Funding page.