How to Debug the Kernel and the Application in Toro

Introduction

In this section, we present three methods to debug the kernel and the user application. We first present how to add debug symbols during the compilation. Then, we present the debug serial console which is the main mechanism to get debug information during runtime. We finish this section by showing how to use GDB together with QEMU to run the kernel step by step and set breakpoints.

Compiling the Kernel with Debug Information

In order to track exceptions, the kernel and the application can be compiled with debug symbols. This allows to get a backtrace if an exception happens. To do this, first set the following checkbox in Lazarus:

Then, edit Toro.inc and define the symbol DebugCrash by uncommenting the line:

{$DEFINE DebugCrash}

To see the backtrace working, you can compile the example ToroException.pas. If everything runs correctly, we get the following message:

Together with the exception crash we get the backtrace pointing the line that made the exception.

TODO: backtrace does not work anymore! Only the adds can be displayed but not the symbols.

Enable Debug Serial Console

Toro relies on the serial console to output debug information. The file Toro.inc contains the symbols that can be enabled to produce debug information. Before to enable any of this symbols, we have to enable the symbol {$DEFINE DEBUG} which initializes serial console. After doing this, we have to configure QEMU to redirect the serial console to a file. To do this, we add the following to the line that invokes QEMU:

-serial file:torodebug.txt

By doing this, we get in the file torodebug.txt the output from the serial console. For example, a typical output may be:

22/10/2017-10:37:33 CPU0 Thread#0 Initialization of debugging console
22/10/2017-10:37:33 CPU0 Thread#0 Crash dumping is Enabled
18/08/2017-09:27:54 CPU0 Thread#272039544 DedicateNetwork: dedicating on CPU0
18/08/2017-09:27:54 CPU0 Thread#272039544 DedicateNetwork: New Driver dedicated to CPU#0

Debugging a user application by using GDB

In this section, we are going to debug an application in Toro that is deployed as a KVM Qemu guest. The application is built in a local host (the development host) and it is executed in the a remote host (deployment host). The development host has Lazarus 2.1.0 with FPC cross-compiled with Linux as the target platform. The deployment host is a Linux Debian 10 with QEMU-KVM installed. The remote host must be set up as it is explained here. In the following steps, we are first going to setup the IDE, second, deploy the application in a remote host as a Toro's guest, and third, debug the application in the remote host by using remote debugging from the Lazarus. We use the exanple HelloWorld as a running example.

Step1. Configure Lazarus and Compile the application

You have to open HelloWorld.lpi and configure Lazarus as shown in the picture:

You have to set the Architecture to i386:x86-64, then, change the Debugger_Remote_Hostname and the Debugger_Remote_Port to your own GDB server. The Debugger_Startup_Options are optional and it could be used for debugging the debugger. Also, you have to set the Internal_Start_Break to GDBNone.

Go to the project's options and set optimization level to "0" or disabled. Then, open Toro.inc and uncomment the following symbol:

{$DEFINE UseGDBstub}     

Also, uncomment line 40 in HelloWorld.pas to include the Gdbstub unit. Finally, you have to compile the application by doing Run>Build. If compilation has been successful, you will find the HelloWorld binary in the HelloWorld directory. You have to copy this binary in the remote host at the HelloWorld directory. To do this, you can use WinSCP or ssh.

Step 2. Launch an instance

You have to launch an instance of the HelloWorld application as a KVM guest. First, comment the lines 50, 53, 58, 59 and 60 of CloudIt.sh to prevent to recompile the kernel. Then, go to HelloWorld directory and execute:

../CloudIt.sh HelloWorld "" "-serial tcp::1234,server"

This is going to launch QEMU and waits until the GDB client connect to the host at port 1234:

QEMU 5.1.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -serial tcp::1234,server: info: QEMU waiting for connection on: disconnected:tcp:0.0.0.0:1234,server

You can connect any GDB client to that port. In the following, we are going to use Lazarus and GDB in Windows host to set some breakpoints and do "step-by-step" execution.

Optional

If you are in Linux and you want to debug the appliance, you can simply run the following command in the HelloWorld directory:

gdb HelloWorld -ex 'set arch i386:x86-64' -ex 'target remote localhost:1234'

Additionally, you can add -ex 'set debug remote 1' to debug the GDB protocol.

Step 3. Launch the debugger

First, we are going to set some breakpoints. Let's put a breakpoint in HelloWorld.pas:51. Then, you have to go to Run>Run to start the debugging session. When the breakpoint is reached, you are going to see something like that:

Then you can execute the next by either pressing on F8 or clicking on step-over:

If we continue clicking on step-over, we execute the example step by step until we end in the halt loop.

To stop the execution, just go to Run>Stop, and then come back to the terminal and press ctrl+c to close QEMU. You can watch a video that explains all the steps to debug the HelloWorld example at here.

Limitations

• Limited number of breakpoints
• The execution can't be paused