DirectProgramming/DPC++/Jupyter/oneapi-essentials-training/11_Intel_Distribution_for_GDB/gdb_oneapi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intel® Distribution for GDB*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will cover using the Intel® Distribution for GDB* to debug oneAPI applications on the GPU.\n",
    "\n",
    "##### Sections\n",
    "- [Intel Distribution for GDB Overview](#Intel-Distribution-for-GDB-Overview)\n",
    "- [How does the Intel Distribution for GDB debug GPUs?](#How-does-Intel-Distribution-for-GDB-debug-GPUs?)\n",
    "- [GDB Commands](#GDB-Commands)\n",
    "- [Debug an Application](#Debug-an-Application)\n",
    "- [Multi-Device Debugging](#Multi-Device-Debugging)\n",
    "\n",
    "Note: Unlike other modules in the oneAPI Essentials series, this notebook is designed for the DevCloud and cannot be run in a local environment. This is because when GDB pauses the GPU execution, display rendering is also interrupted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "The goal of this notebook is to show how the Intel® Distribution for GDB* can help you debug GPU kernels. At the end of module, you will be able to:\n",
    "<ul>\n",
    "    <li>Run the Intel Distribution for GDB.</li>\n",
    "    <li>Understand inferiors, threads, and SIMD lanes as shown in GDB.</li>\n",
    "    <li>Use different methods to examine local variables for different threads and lanes.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intel Distribution for GDB Overview\n",
    "\n",
    "Intel® Distribution for GDB* (*gdb-oneapi* executable) is part of the Intel® oneAPI Base Toolkit. It can be used to debug oneAPI applications written in several different languages targeting various different accelerator devices.\n",
    "\n",
    "<img src=\"assets/gdb_overview.jpg\">\n",
    "\n",
    "### Major Features\n",
    "* Multi-target: The debugger can orchestrate multiple targets for different architectures. This feature allows you to debug the \"host\" portion and the \"kernel\" of a DPC++ program in the same GDB* session.\n",
    "* Auto-attach: The debugger automatically creates an inferior that attaches itself to the Intel® Graphics Technology target to be able to receive events and control the GPU for debugging.\n",
    "* Thread and SIMD lanes: The debugger displays SIMD lane information for the GPU threads on the command line interface. You can switch among active threads and lanes.\n",
    "* Support for debugging a kernel offloaded to a CPU, GPU, or FPGA-emulation device.\n",
    "\n",
    "\n",
    "## How does the Intel Distribution for GDB debug GPUs?\n",
    "\n",
    "\n",
    "### Compilation and Execution for Debug\n",
    "When debugging oneAPI applications with gdb-oneapi, debug information for the GPU needs to be generated and embedded in the application. The compilation and execution process looks like the following.\n",
    "\n",
    "<img src=\"assets/gpu_debug.jpg\">\n",
    "\n",
    "1. Source code is compiled. Host code is compiled normally while kernel code is compiled with debug info into SPIR-V intermediate representation format embedded in the host binary.\n",
    "    * Use -g (generate debug info) and -O0 (disable optimization) compiler options to debug source.\n",
    "    * May use -O2 to debug optimized code at assembly level.\n",
    "    * Use same optimization level when linking, if compiling and linking separately.\n",
    "    * Ahead-of-time (AOT) compilation also works with GPU debug and can be utilize to avoid JIT compilation everytime application is run.\n",
    "2. Launch appliction with `gdb-oneapi`\n",
    "    * `gdb-oneapi <your_app>`\n",
    "3. Application runtime compiles SPIR-V and debug info into ELF and DWARF formats.\n",
    "4. GPU kernel code is executed and debugged.\n",
    "\n",
    "### Inferiors for GPUs\n",
    "\n",
    "GDB creates objects called *inferior*s to represent the state of each program execution. An inferior usually corresponds to a debugee process. For oneAPI applications, GDB will create one inferior for the native host target and additional inferiors for each GPU or GPU tile. When a GPU application is debugged, the debugger, by default, automatically launches a `gdbserver-gt` process to listen to GPU debug events. The `gdbserver-gt` target is then added to the debugger as an inferior.\n",
    "\n",
    "<img src=\"assets/gdb_gpu_inferior.jpg\">\n",
    "\n",
    "To see information about the inferiors while debugging. Use the `info inferiors` GDB command.\n",
    "\n",
    "### Debugging Threaded GPU SIMD Code\n",
    "\n",
    "GPU kernel code is written for a single work-item. When executing, the code is implicitly threaded and widened to vectors of work-items. In the Intel Distribution for GDB, variable locations are expressed as functions of the SIMD lane. The lane field is added to the thread representation in the form of `<inferior>.<thread>:<lane>`.\n",
    "\n",
    "Users can use the `info threads` command to see information about the various active threads. The `thread` command can be used to switching among active threads and SIMD lanes. The `thread apply <thread>:<lane> <cmd>` command can be used to apply the specified command to the specified lanes.\n",
    "\n",
    "SIMD Lanes Support:\n",
    "\n",
    "    * Only enabled SIMD lanes are displayed\n",
    "    * SIMD width is not fixed\n",
    "    * User can switch between enabled SIMD lanes\n",
    "    * After a stop, GDB switches to an enabled SIMD lane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GDB Commands\n",
    "\n",
    "The following table lists some common GDB commands. If a command has special functionality for GPU debugging, description will be shown in orange. You may also consult the [Intel Distribution for GDB Reference Sheet](https://software.intel.com/content/www/us/en/develop/download/gdb-reference-sheet.html).\n",
    "\n",
    "| Command | Description |\n",
    "| ---: | :--- |\n",
    "| help \\<cmd> | Print help information. |\n",
    "| run [arg1, ... argN] | Start the program, optionally with arguments. |\n",
    "| break \\<file>:\\<line> | Define a breakpoint at a specified line. |\n",
    "| info break | Show defined breakpoints. |\n",
    "| delete \\<N> | Remove Nth breakpoint. |\n",
    "| step / next | Single-step a source line, stepping into / over function calls. |\n",
    "| info args/locals | Show the arguments/local variables of the current function. |\n",
    "| print \\<exp> | Print value of expression. |\n",
    "| x/\\<format> \\<addr> | Examine the memory at \\<addr>. |\n",
    "| up, down | Go one level up/down the function call stack |\n",
    "| disassemble | Disassemble the current function.  <font color='orange'> If inside a GPU kernel, GPU instructions will be shown. </font> |\n",
    "| backtrace | Shown the function call stack. |\n",
    "| info inferiors | Display information about the inferiors. <font color='orange'> GPU debugging will display additional inferior(s) (gdbserver-gt). </font> |\n",
    "| info threads \\<thread> | Display information about threads, including their <font color='orange'> active SIMD lanes. </font> |\n",
    "| thread \\<thread>:\\<lane> | Switch context to the <font color='orange'> SIMD lane of the specified thread. <font> |\n",
    "| thread apply \\<thread>:\\<lane> \\<cmd> | Apply \\<cmd> to specified lane of the thread. |\n",
    "| set scheduler-locking on/step/off | Lock the thread scheduler. Keep other threads stopped while current thread is stepping (step) or resumed (on) to avoid interference. Default (off). |\n",
    "| set nonstop on/off | Enable/disable nonstop mode. Set before program starts. <br> (off) : When a thread stops, all other threads stop. Default. <br> (on) : When a thread stops, other threads keep running. |\n",
    "| print/t $emask | Inspect the execution mask to show active SIMD lanes. | "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debug an Application\n",
    "The kernel we're going to debug is a simple array transform function where the kernel adds 100 to even elements of the array and sets the odd elements to be -1. Below is the kernel code, the entire source code is [here](src/array-transform.cpp).\n",
    "``` cpp\n",
    "54        h.parallel_for(data_range, [=](id<1> index) {\n",
    "55            size_t id0 = GetDim(index, 0);\n",
    "56            int element = in[index]; // breakpoint-here\n",
    "57            int result = element + 50;\n",
    "58            if (id0 % 2 == 0) {\n",
    "59                result = result + 50; // then-branch\n",
    "60            } else {\n",
    "61                result = -1; // else-branch\n",
    "62            }\n",
    "63            out[index] = result;\n",
    "64        });\n",
    "```\n",
    "\n",
    "### Compile the Code\n",
    "Execute the following cell to compile the code. Notice the compiler options used to disable optimization and enable debug information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! dpcpp -O0 -g src/array-transform.cpp -o bin/array-transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a debug script\n",
    "To debug on the GPU, we're going to write the GDB debug commands to a file and then submit the execution of the debugger to a node with GPUs.\n",
    "\n",
    "In our first script, we'll get take a look at how inferiors, threads, and SIMD lanes are represented. Our debug script will perform the following tasks. \n",
    "1. Set a temporary breakpoint in the DPCPP kernel at line 59.\n",
    "2. Run the application in the debugger.\n",
    "3. Display information about the active inferiors once the breakpoint is encountered.\n",
    "4. Display information about the active threads and SIMD lanes.\n",
    "5. Display the execution mask showing which SIMD lanes are active.\n",
    "6. Remove breakpoint.\n",
    "7. Continue running.\n",
    "\n",
    "Execute the following cell to write the debug commands to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile lab/array-transform.gdb\n",
    "#Set Breakpoint in the Kernel\n",
    "echo ================= (1) tbreak 59 ===============\\n\n",
    "tbreak 59\n",
    "\n",
    "# Run the application on the GPU\n",
    "echo ================= (2) run gpu ===============\\n\n",
    "run gpu\n",
    "\n",
    "echo ================= (3)  info inferiors ============\\n\n",
    "info inferiors\n",
    "\n",
    "echo ================= (4) info threads ============\\n\n",
    "info threads\n",
    "\n",
    "# Show execution mask that show active SIMD lanes.\n",
    "echo ================= (5) print/t $emask ============\\n\n",
    "print/t $emask\n",
    "\n",
    "echo ================= (6) c ==========================\\n\n",
    "c "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start the Debugger\n",
    "The [run_debug.sh](run_debug.sh) script runs the *gdb-oneapi* executable with our debug script on the compiled application.\n",
    "\n",
    "Execute the following cell to submit the debug job to a node with a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! chmod 755 q; chmod 755 run_debug.sh; if [ -x \"$(command -v qsub)\" ]; then ./q run_debug.sh; else ./run_debug.sh; fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation of Output\n",
    "1. You should see breakpoint 1 created at line 59.\n",
    "2. Application is run with the *gpu* argument to execute on the GPU device. Program should stop at the kernel breakpoint.\n",
    "3. With context now automatically switched to the device. The *info inferiors* command will show the active GDB inferior(s). Here, you should see two, one corresponds to the host portion, another, the active one, for gdbserver-gt which is debugging the GPU kernel. \n",
    "4. The *info threads* command allows you to examine the active threads and SIMD lanes. There should be 8 threads active. Notice that only even SIMD lanes are active, this is because only the even work-items encounter the breakpoint at line 59.\n",
    "5. Printing the $emask execution mask also shows the even lanes being active.\n",
    "6. Continue running the program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debug the Application Again\n",
    "\n",
    "Now, we will debug the application again. This time, we'll switch threads, use the scheduler-locking feature, and print local variables.\n",
    "Run the following cell to write new GDB commands to array-transform.gdb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile lab/array-transform.gdb\n",
    "#Set Breakpoint in the Kernel\n",
    "echo ================= (1) break 59 ===============\\n\n",
    "break 59\n",
    "echo ================= (2) break 61 ===============\\n\n",
    "break 61\n",
    "\n",
    "# Run the application on the GPU\n",
    "echo ================= (3) run gpu ===============\\n\n",
    "run gpu\n",
    "\n",
    "# Keep other threads stopped while current thread is stepped\n",
    "echo ================= (4) set scheduler-locking step ===============\\n\n",
    "set scheduler-locking step \n",
    "\n",
    "echo ================= (5) next ===============\\n\n",
    "next \n",
    "\n",
    "echo ================= (6) info threads 2.* ===============\\n\n",
    "info threads 2.*\n",
    "\n",
    "echo ================= (7) Print element ============\\n\n",
    "print element\n",
    "\n",
    "# Switch thread\n",
    "echo ================= (8) thread 2.1:5 =======================\\n\n",
    "thread 2.1:4\n",
    "\n",
    "echo ================= (9) Print element ============\\n\n",
    "print element\n",
    "\n",
    "echo ================= (10) thread apply 2.1:* print element =======================\\n\n",
    "thread apply 2.1:* print element\n",
    "\n",
    "# Inspect vector of a local variable, 8 elements, integer word\n",
    "echo ================= (11) x/8dw &result =======================\\n\n",
    "x /8dw &result\n",
    "\n",
    "echo ================= (12) d 1 =======================\\n\n",
    "d 1\n",
    "echo ================= (13) d 2 =======================\\n\n",
    "d 2\n",
    "echo ================= (14) c ==========================\\n\n",
    "c "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Debugger Again To Examine Variables, Memories\n",
    "Run the following cell to run the debugger for the second time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! chmod 755 q; chmod 755 run_debug.sh; if [ -x \"$(command -v qsub)\" ]; then ./q run_debug.sh; else ./run_debug.sh; fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explanation of Output\n",
    "1. Set Breakpoint at line 59 for the even lanes.\n",
    "2. Set Breakpoint at line 61 for the odd lanes.\n",
    "3. Start the application, it will stop at breakpoint 2. At this point all the threads should be stopped and active for the odd SIMD lanes.\n",
    "4. Set schedule-locking so that when the current thread is stepped all other threads remain stopped.\n",
    "5. Step the current thread (thread 2.1), breakpoint at line 59 is encountered only for current thread.\n",
    "6. Show the threads and where each thread is stopped. Notice the current thread is stopped and active at the even SIMD lanes.\n",
    "7. Print local variable element.\n",
    "8. Switch to a different lane.\n",
    "9. Print local variable element again, this time you should see a different value.\n",
    "10. Use thread apply to print element for all lanes of the 2.1 thread.\n",
    "11. Print vectorized result.\n",
    "12. Delete breakpoint.\n",
    "13. Delete breakpoint.\n",
    "14. Run until the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Device Debugging\n",
    "\n",
    "The Intel Distribution for GDB can debug applications that offload a kernel to multiple GPU devices. Each GPU device appear as a separate inferior within the debugger. Users can switch to the context of a thread that corresponds to a particular GPU or CPU using the `inferior <id>` command. Threads of the GPUs can be independently resumed and the thread state can be individually examined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "* [Intel Distribution for GDB Landing Page](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-for-gdb.html)\n",
    "* [Intel Distribution for GDB Release Notes](https://software.intel.com/content/www/us/en/develop/articles/gdb-release-notes.html)\n",
    "* [Intel Distribution for GDB Reference Sheet](https://software.intel.com/content/www/us/en/develop/download/gdb-reference-sheet.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "  * Used Intel Distribution for GDB to debug a GPU application.\n",
    "  * Used various-GPU related GDB commands."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 (Intel® oneAPI)",
   "language": "python",
   "name": "c009-intel_distribution_of_python_3_oneapi-beta05-python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "310.667px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}