Unable to debug python code in a C++ application that embeds the python interpreter

[blargelJim]

Im trying to debug a simple C++ application that embeds python and runs a python script.
I try to use attach to process to start debugging the C++ application and while it does attach itself to the process (if I choose python native) it never stops at any of the breakpoints.

This is what I've tried so far:

1. Start the C++ app (the C++ app will wait for user input before loading the python file)
2. Start Visual Studio (2022), continue without code, then open the python script that the C++ app will load and set a breakpoint in it.
3. Debug->Attach to process (Python (native) code):
   
4. Type the input into the C++ app so it will proceed and load and run the python code.
   -> No break points are triggered.

It works fine if I want to use mixed-mode debugging, both the breakpoints in C++ and in the python code are triggered and I can step through the code in both C++ and python. But trying to debug ONLY the python code dose'nt which seems strange.

I've attached the C++ application as a file as well as the python code. Its sitting here:
source.zip
I've been testing this out with python 3.9.7
The only project settings I've set for my C++ project is to add the python include path as well as point out python library to the linker (additional dependencies)
To run the C++ application you need to pass some program arguments, in my case: "multiply multiply 3 2" this will call the multiply function in the multiply python module with arguments 2 and 3.
As it was a bit extra work to update pythons sys.path to find the multiply module I simply put it in the root folder where I have python installed (where the python interpreter, 'python.exe', is located).

[int19h]

If you're trying to do pure Python debugger, you need to use "Python code". The "Python (native) code" debugger is meant for mixed-mode only (unfortunately, we couldn't make it transparently select the correct debugger as it used to be in older VS versions).

[blargelJim]

Hi @int19h and thank you for the quick feedback.

I've tested this out and this causes the C++ app to report back:
Py_IsInitialized returned false.
Error when injecting code in target process. Error code (on windows): 2

Im not sure what is outputting the feedback.
Visual studio then tells me that something went wrong:

In visual studio, the output from the output window (Debug) says:
The program '[5456] TestDebugPyFromCpp.exe' has exited with code 4294967295 (0xffffffff).

The output for pylance says:
Info: (5600) Pylance language server 2023.7.10 (pyright 74d8f3c4) starting
Info: (5600) Server root directory: c:\program files\microsoft visual studio\2022\community\common7\ide\extensions\microsoft\python\core\pylance\dist
Info: (5600) Starting service instance ""
Info: (5600) No source files found.
Info: (5600) Background analysis(1) root directory: c:\program files\microsoft visual studio\2022\community\common7\ide\extensions\microsoft\python\core\pylance\dist
Info: (5600) Background analysis(1) started
Info: (5600) Indexer background runner(2) root directory: c:\program files\microsoft visual studio\2022\community\common7\ide\extensions\microsoft\python\core\pylance\dist (index)
Info: (5600) Indexing(2) started
Info: (5600) scanned(2) 0 files over 0 exec env
Info: (5600) Indexing finished(2).

I've been looking at this issue report: https://github.com/microsoft/debugpy/issues/887 but not sure if its the same problem? That issue seems to be reported with VS code, Im running visual studio 2022.

[int19h]

The message is coming from the injector process that does the necessary magic to get the pure Python debugger up and running in a non-cooperative process. Ultimately what it does is basically running debugpy.connect() for you on its own thread, but since that is Python code, it needs Python to be there and not blocked on GIL in order to do so. Thus if you try to attach before Python is loaded & initialized in the process, it will fail.

Mixed-mode is not affected by this because the debugger there is completely different, and that one directly operates on internal Python data structures rather than working via Python APIs (this is why it's not available on newer Python versions; too much has changed there internally and we couldn't keep up), so it can attach to any process, even one that doesn't have Python DLLs loaded yet.

So, for the time being, attaching using pure Python debugger to a process that hasn't loaded & initialized Python yet is an unsupported scenario. This might change in the future, but for now, try pausing your C++ app between the initialization and actually executing code.

BTW, if you're changing the source anyway, it might be easier to add debugpy.connect(...) or debugpy.listen(...); debugpy.wait_for_client() invocations directly at the beginning of the Python code you're trying to run (or maybe run them first in a separate eval). This is generally more reliable than attach-to-PID, because the debuggee is cooperative in this scenario.

[blargelJim]

I've updated the code so that the debugger is able to attach to the process with the debug profile set to "Python" (i.e. not "native python" as before). I did this by adding a function that releases the GIL and then calls sleep in a loop a couple of times that should allow the debugger to its thing.
That function looks like this:

The entire code is here
source.zip
The setup is the same, the python module multiply.py should be placed somewhere where python can find it and run it.

That seems to work fine, Im now able to start the C++ app that embeds python and attach to it using the "Attach to process".
However, it now seems like another issue crops up

As soon as I've attached the debugger to the process I get a print out from the C++ app:
The threading module was not imported by user code in the main thread. The debugger will attempt to work around https://bugs.python.org/issue37416.
And then as soon as the C++ app calls the python code that I want to debug I seem to be getting an exception from python:
Exception ignored in: <module 'threading' from 'D:\Python\Python397\Lib\threading.py'>
Traceback (most recent call last):
File "D:\Python\Python397\Lib\threading.py", line 1428, in _shutdown
assert tlock.locked()
AssertionError:

This seems to refer to a bug that should be resolved in versions above 3.7, however Im seeing it in 3.9 anyways (the issue being microsoft/ptvsd#1542)

I looked at https://bugs.python.org/issue37416 and your comment "I think that a similar problem can also occur in an embedded Python scenario with multithreading. Consider what happens if the hosted interpreter is initialized from the main thread of the host app - but some Python code is then run from the background thread, and that code happens to be the first in the process to import threading. Then that background thread becomes the "main thread" for threading, with the same results as described above."
Perhaps this is what happens? The example code I've setup dose'nt use any threads on its own, but Im not sure about the python C-API calls that it makes, perhaps some of those do?
In my example, I only call Py_Initialize and then wait for the user input (in here I do use Py_BEGIN_ALLOW_THREADS and PyGILState_Check). I would'nt expect any of these calls to do anything that runs python code but cant say for sure.

As for the debugpy - I had a quick look at using that but I could'nt find much documentation on how to use it. Does it work in visual studio? The end goal is to be able to use visual studio for debugging the python code, I was'nt sure if that is what this lib does?

[axelande]

Could the symlinks be an issue here? I had huge problems but finally solved it: https://stackoverflow.com/questions/74190153/no-symbols-loaded-for-c-in-mixed-debugging-from-python/74486282#74486282