Debugging python and C++ code when using pybind11

[JakobThumm]

Bug type: Debugger

• OS and Version: Ubuntu 20.04
• VS Code Version: 1.64.2 x64
• C/C++ Extension Version: 1.8.4
• Other extensions you installed: python v2022.0.18

Describe the bug
I want to run and debug my own C++ extensions for python in "hybrid mode" in VSCode. Since defining your own python wrappers can be quite tedious, I want to use pybind11 to link C++ and python. I love the debugging tools of vscode, so I would like to debug both my python scripts as well as the C++ functions in vscode.

Fortunately, debugging python and C++ files simultaneously is possible by first starting the python debugger and then attach a gdb debugger to that process as described in detail in nadiah's blog post (Windows users, please note this question). This works fine for me. Unfortunately, they define the C++ -- python bindings manually. I would like to use pybind11 instead.

I created a simplified example that is aligned with nadiah's example using pybind11. Debugging the python file works but the gdb debugger doesn't stop in the .cpp file. According to this github question it should be theoretically possible but there are no details on how to achieve this.

To Reproduce
Here I try to follow nadiahs example as closely as possible but include pybind11 wrappers.

Setting up the package

Create a virtual environment (also works with anaconda, as described below)

virtualenv --python=python3.8 myadd
cd myadd/
. bin/activate

Create file myadd.cpp

#include <pybind11/pybind11.h>

float method_myadd(float arg1, float arg2) {
    float return_val = arg1 + arg2;
    return return_val;
}

PYBIND11_MODULE(myadd, handle) {
    handle.doc() = "This is documentation";
    handle.def("myadd", &method_myadd);
}

, myscript.py

import myadd

print("going to ADD SOME NUMBERS")

x = myadd.myadd(5,6)

print(x)

and setup.py

from glob import glob
from distutils.core import setup, Extension
from pybind11.setup_helpers import Pybind11Extension

def main():
    setup(name="myadd",
          version="1.0.0",
          description="Python interface for the myadd C library function",
          author="Nadiah",
          author_email="nadiah@nadiah.org",
          ext_modules=[Pybind11Extension("myadd",["myadd.cpp"])],
          )


if __name__ == "__main__":
    main()

Clone the pybind11 repo

git clone git@github.com:pybind/pybind11.git

and install the python package

pip install pybind11

Run the setup script

python3 setup.py install

Now, we can already run the python script

python myscript.py

Setting up vscode

Open vscode

code .

Select the python interpreter with Ctrl+Shift+p -> Select python interpreter -> ./bin/python, now in the lower bar, you should see virtualenv myadd.
Create the launch.json file by clicking the debug symbol and 'Create new launch configuration'.
This is my launch.json (This might be the problem)

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python",
            "type": "python",
            "request": "launch",
            "program": "myscript.py",
            "console": "integratedTerminal"
        },
        {
            "name": "(gdb) Attach",
            "type": "cppdbg",
            "request": "attach",
            "program": "${workspaceFolder}/bin/python", /* My virtual env */
            "processId": "${command:pickProcess}",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ],
            "additionalSOLibSearchPath": "${workspaceFolder}/build/lib.linux-x86_64-3.8;${workspaceFolder}/lib;${workspaceFolder}/lib/python3.8/site-packages/myadd-1.0.0-py3.8-linux-x86_64.egg/"
        }
    ]
}

Note that I added the "additionalSOLibSearchPath" option in accordance to the github question but it did not change anything.

Debugging

In vscode, add breakpoints in myscript.py in line 5 and 7, and in myadd.cpp in line 5.
Now, first start the python debugger and let it stop on the breakpoint in line 5.
Then go to a terminal and get the correct process id of the running python script.

ps aux | grep python

The second to last process is the correct one in my case. E.g.

username      **65715**  3.0  0.0 485496 29812 pts/3    Sl+  10:37   0:00 /home/username/myadd/bin/python /home/username/.vscode/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy --connect 127.0.0.1:38473 --configure-qt none --adapter-access-token ... myscript.py

In this example, 65715 would be the correct process id.
Now, in vscode start the (gdb) Attach debugger and type in the process id in the search bar. Hit enter, now you need to type y in the console to allow the attaching and type in your sudo password.
If you are following nadiah's example, you can now press continue on the python debug bar and the script will stop on the C++ breakpoint.
For this pybind11 example, the script does not stop on the C++ breakpoint.

Project Structure

Your project structure now should look like this

myadd
| bin/
| build/
| dist/
| lib/
| myadd.cpp
| myadd.egg-info/
| myscript.py
| pybind11/
| setup.py

Additional context
As stated in the github post, one has to ensure that the debug flag is set. Therefore, I added a setup.cfg file

[build_ext]
debug=1
[aliases]
debug_install = build_ext --debug install

And ran

python setup.py debug_install

but this did not help as well.

[WardenGnaw]

As we are primary focused on C/C++ development, we are looking for community to help resolve this issue.

[hwiedPro]

Hi,
if you have gdb and python3-dbg installed.
You can simply use gdb.
If you are working in an virtualenv you have to link /usr/bin/python3-dbg into your env's bin dir and then run your script with python-dbg.

[benibenj]

To avoid manually launching both the Python and C++ debuggers, you can use the "Python C++ Debugger" extension which automates this process :)
https://marketplace.visualstudio.com/items?itemName=benjamin-simmonds.pythoncpp-debug

[kopalx]

Hi,
I have the Ubuntu 22.04 on my desktop and the Rocky Linux on our server.
The debug as described above runs on the Rocky, but I get a crush (core dump ###) on the desktop.
After a crush, VSCODE deletes all logs.

How can I debug the crash?

thanks in advance,
Alex

[wangyuelucky]

hi @JakobThumm Any updates on this? Looks like i face the same issue.

[JakobThumm]

I was not able to get a successful debugging running.
I tried most of the recommendations of the community here and in other forums but I always ran into different issues.
For me, it was less time to write a large number unit tests for the C++ code than to get a combined debugging running.

[wangyuelucky]

thanks all the same.

[wangyuelucky]

Hi guys, This worked well for me at last.
set the env variable before execute setup.py, then vscode debugger stop at the breakpoints.
``
export CPPFLAGS=-g3
python3 setup.py debug_install