For Python debugging to work, a specific Python toolchain MUST be set

[nikhilm]

Description of the bug:

I spent several frustrating hours on this, so I just want to leave a note for others. I was unable to find anything else on the internet. #2093 is not the same thing.

For those looking for the fix
Python code debugging does not work if you rely on the Bazel to autodetect the Python toolchain installed on your system!. This assumes you are using rules_python. At the minimum, you must explicitly specify a non-portable, non-hermetic, system toolchain like so. Obviously, it is recommended to create a better toolchain for production use cases (example):

// In your top-level BUILD.bazel
py_runtime(
    name = "python3.10",
    interpreter_path = "/usr/bin/python3.10",
    python_version = "PY3",
)

py_runtime_pair(
    name = "py3.10",
    py3_runtime = ":python3.10",
)

toolchain(
    name = "system-py3",
    toolchain = ":py3.10",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

// In WORKSPACE
register_toolchains("//:system-py3")

If you only care about the fix, you can stop reading here.

The root cause is the various shims that interpose and create child processes. By default, a py_binary or py_test target's actual executable file is this wrapper. Without a toolchain, PYTHON_BINARY is set to py3wrapper.sh, a shell script. Even when pydevd is launched by the IDE in multiprocess mode, on every process launch, it tries to determine if the new process is a Python process. If the name isn't python or similar, it then looks for a shebang line to be python, which also fails in this case. So it never attaches to py3wrapper.sh or any subprocesses created by it (so the actual running program).

I had to resort to modifying the included copy of pydevd in the IDE (Python plugin) to switch it to debug mode and figure this out.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Check out nikhilm/bazel-python-example@fef3300 which does not have a toolchain specified.
2. Set a breakpoint at any line in greeting() or foo().
3. Try to run example_test.py with Debugging via the Bazel integration.
4. Debugger does not stop at any breakpoints.

After adding a toolchain

1. Check out nikhilm/bazel-python-example@8f2dd48 which adds a toolchain that works on Ubuntu 22.04. You may need to tweak the path to the python binary.
2. Same as above
3. Same as above
4. Debugger should stop at the breakpoint.

Which Intellij IDE are you using? Please provide the specific version.

Tested on CLion 2023.1, IntelliJ IDEA 2023.1 (Community Edition) Build #IC-231.8109.175

What programming languages and tools are you using? Please provide specific versions.

Bazel 6.0.0

What Bazel plugin version are you using?

Development tip 9a7b789

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

No response

[tpasternak]

It is indeed true, thank you for the detailed analysis!

[tpasternak]

https://youtrack.jetbrains.com/issue/PY-60259