Skip to content

Advanced course: Debugging the Linux kernel via KGDB and a serial cable (contains PC to PC and also embedded Yocto over RS 232)

Roelof Berg edited this page Apr 17, 2020 · 2 revisions

Debugging the Linux Kernel via a Nullmodem-Cable:

It took me a while to get a kgdb connection with Linux kernel awareness fully running. I share my way of doing this with Ubuntu Eoan (optional: Yocto Warrior) in 2020 here:

Tested with:

  • Debugging a linux based Intel PC from an Intel MacBook running MacOS Catalina. Using the gdb from the Homebrew package "i386-elf-gdb“. (wituout „-tui“ option in GDB)
  • Debugging a linux based ARM target (i.mx6, Yocto) from a linux based Intel PC.

Prerequisites:

You need two computers and a serial nullmodem cable. Check the cable by firiing up a serial termianl (e.g. screen or putty) on both hosts, connecting to your serial port (e.g. /dev/ttyS0 or /dev/ttyUSB0) and print characters from each station to the other. Remember the /dev/tty ports you confirmed.

Preparation:

You need on the first debuggee computer, we call it „target":

  • Special kernel installed that contains symbols, kgdb support etc.

  • Learn how to compile and install a kernel and use in make menuconfig belows configuration. You can search for Sybmbols with F8 or the / key in menuconfig. (E.g. wiki.ubuntu.com. There take care in the first paragraph to execute deb-src before apt-get :)

     # CONFIG_SERIAL_KGDB_NMI is not set
     CONFIG_CONSOLE_POLL=y
     # CONFIG_DEBUG_INFO is not set
     CONFIG_KGDB=y
     CONFIG_KGDB_SERIAL_CONSOLE=y
     # CONFIG_KGDB_TESTS is not set
     # CONFIG_KGDB_KDB is not set
     CONFIG_FRAME_POINTER=y
     CONFIG_DEBUG_INFO=y
     # CONFIG_DEBUG_INFO_REDUCED is not set
     # CONFIG_DEBUG_INFO_SPLIT is not set
     CONFIG_DEBUG_INFO_DWARF4=y
     CONFIG_GDB_SCRIPTS=y
     CONFIG_STRIP_ASM_SYMS=y
     # CONFIG_RANDOMIZE_BASE is not set
    

(Note for advanced Yocto use, skip if you're debugging a PC: In yocto I created in my layer a file: recipes-kernel/linux/linux-mainline_%.bbappend with the content: FILESEXTRAPATHS_prepend := "${THISDIR}/files:" SRC_URI += "file://kgdb.cfg“ And in files/kgdb.cfg I added the config fragment shown above (without the on ARM unavailable options CONFIG_RANDOMIZE_BASE and CONFIG_FRAME_POINTER) )

You need on the second debugger computer, we call it „debugger pc":

  • Full kernel source code, same code you used to compile the kernel above. (If you compiled the .o and .ko objects in place and not in a build-folder you better not copy the directory from the other pc, where you called make etc. in, but then better grab fresh sources again.)
  • vmlinux file containing the symbols (lies in the kernel source root, or build folder on the highest level after kernel make).
  • vmlinux-gdb.py file that was made during the kernel build (also lies at the same position on the highest level.).
  • All scripts in the folder scripts/gdb (Folder scripts in the same toplevel-position. If you use a dedicated build folder use the script folder from there, not from the source folder.)
  • (Advanced: If both computers don’t match in CPU, like Intel and Arm, a cross-gdb build. Ignore if you're on Intel/AMD.)

Note for advanced Yocto use, I did something like (ignore if you debug a PC):

 bitbake -c patch virtual/kernel     #(apply the changed kernel config from above)
 bitbake -f -c compile virtual/kernel   #(unpack is not sufficient because of vmlinux-gdb.py)
 mkdir ~/gdbenv
 cp -a tmp/work-shared/phyboard-mira-imx6-14/kernel-source/. ~/gdbenv
 cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux ~/gdbenv
 cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux-gdb.py ~/gdbenv
 mkdir ~/gdbenv/scripts
 cp -r tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/scripts/gdb ~/gdbenv/scripts

Then (ignore if you're on a PC) yocto bitbake -c populate_sdk [my-image]

Then (still ignore on PC) install the sdk .sh-installation file from your deploy directory on the debugger pc and start the environment as guided by the output of the install script (remember that command), then use "$GDB" for starting the cross-gdb instead of „gdb".

Debug execution

Launch on the debugger two console screens:

Console 1, ssh: +++++++++++++++++++++++++++++++++++++++

 ssh user@192.168.x.y
 sudo -s
 echo ttyS0,9600n8 > /sys/module/kgdboc/parameters/kgdboc
 echo 1 > /proc/sys/kernel/sysrq
 echo g > /proc/sysrq-trigger

Console 2, local: ++++++++++++++++++++++++++++++++++++++++

 cd ~/gdbenv
 gdb -tui ./vmlinux

 add-auto-load-safe-path ~/gdbenv
 source ~/gdbenv/vmlinux-gdb.py
 set serial baud 9600
 target remote /dev/ttyS0                        (use the tty port you confirmed in the beginning)
 b [name of the c funtion you want to debug]
 cont

Back to console 1, ssh: +++++++++++++++++++++++++++++++++++++++

 [Now trigger the function, e.g. sudo modprobe yourFancyKernelModule]

Back to console 2, local: ++++++++++++++++++++++++++++++++++++++++

Now use gdb functions, like bt, step, next, finish ...

You can also use linux-aware commands. Call "apropos lx“ in gdb for a list of commands.