Debug Python within a container Couldn't spawn debuggee

[mhlg]

I'm following the instructions as for https://code.visualstudio.com/docs/containers/debug-python

this used to work before upgrading docker to version 19.03.13

Python 3.7.6
Docker version 19.03.13, build 4484c46d9d

Create a project folder
mkdir my-project && cd my-project

Create a minimal python file named app.py
`
def main():
print('HELLO')

if name == "main":
main()
`

Create a python3 virtual environment.
python3 -m venv py-venv

setting.json
{ "python.pythonPath": "py-venv/bin/python" }

tasks.json
{ "version": "2.0.0", "tasks": [ { "type": "docker-build", "label": "docker-build", "platform": "python", "dockerBuild": { "tag": "pydockerdebug:latest", "dockerfile": "${workspaceFolder}/Dockerfile", "context": "${workspaceFolder}", "pull": true } }, { "type": "docker-run", "label": "docker-run: debug", "dependsOn": [ "docker-build" ], "python": { "file": "app.py" } } ] }

launch.json
{ "configurations": [ { "name": "Docker: Python - General", "type": "docker", "request": "launch", "preLaunchTask": "docker-run: debug", "python": { "pathMappings": [ { "localRoot": "${workspaceFolder}", "remoteRoot": "/app" } ], "projectType": "general" } } ] }

Create a Dockerfile
`
FROM python:3.8-slim-buster
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app
ADD . /app

RUN useradd appuser && chown -R appuser /app
USER appuser

CMD ["python", "app.py"]
`

Run Debug

and the following error occurs

Couldn't spawn debuggee: [Errno 2] No such file or directory: '/Users/.../py_docker_debug/py-venv/bin/python'

Command line:['/Users/.../py_docker_debug/py-venv/bin/python', '/debugpy', '--connect', 'host.docker.internal:52703', '--configure-qt', 'auto', '--adapter-access-token', '875b8dec0efaabdcdfad7028575706dbb916c2d8bde6e3fd8c25da2142db2a8a', 'app.py']

[bwateratmsft]

Thanks for the detailed info! Does this work if you don't use a virtual environment?

[mhlg]

Thanks for the detailed info! Does this work if you don't use a virtual environment?

Tried now same problem now the
No such file or directory just points to another python path

Couldn't spawn debuggee: [Errno 2] No such file or directory: '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'

1 More thing
I can see the pydockerdebug:latest container is still running after the error is raised

[bwateratmsft]

Hm, it looks like this issue might actually be in either debugpy or the Python extension. My gut is debugpy since that's the only place I can find the --adapter-access-token argument, so I assume it's there that this process is being launched. Can you open this in their repo? https://github.com/microsoft/debugpy

[mhlg]

Hm, it looks like this issue might actually be in either debugpy or the Python extension. My gut is debugpy since that's the only place I can find the --adapter-access-token argument, so I assume it's there that this process is being launched. Can you open this in their repo? https://github.com/microsoft/debugpy

Sure thanks!

[bwateratmsft]

I'll close this as external to microsoft/debugpy#454, but we can certainly reopen this issue if it ends up being on our side!

[int19h]

@bwateratmsft The command line that it's trying and failing to spawn is:

'/Users/.../py_docker_debug/py-venv/bin/python',
'/debugpy',
'--connect',
'host.docker.internal:52703',
'--configure-qt',
'auto',
'--adapter-access-token',
'875b8dec0efaabdcdfad7028575706dbb916c2d8bde6e3fd8c25da2142db2a8a',
'app.py'

Note the path to Python. Looks like it's trying to use the host path inside the guest, and failing. I believe this is called by this change: microsoft/debugpy#446.

The short story is that before the change, when the debugpy launcher spawned the debuggee, it used sys.executable to determine the path to Python binary to do so. Thus, your launcher forwarding script only had to make sure that the command line generated for the launcher is correct, by rewriting the path to interpreter there, and assumed that it would flow down from there, since "python" or "pythonPath" settings were already applied to produce the original command line.

After the change, the launcher disregards sys.executable, and looks at "python" / "pythonPath" directly. Thus, when you do the forwarding, you need to make sure that all interpreter paths are patched up - not just the launcher command line, but also "pythonPath" in the debug configuration.

[bwateratmsft]

@int19h How though? We don't know the path for Python in the container, and indeed we should not have to look for it.

In general I don't see how this change can work with remote debugging.

[int19h]

The launcher forwarding script does it when it rewrites the command line - looks like it just assumes that Python is on PATH, and therefore it can use unqualified python3. The "pythonPath" setting in debug configuration can be the same - it doesn't have to be a full path, it just needs to be something that can be exec'd.

[bwateratmsft]

So in that case it seems like it should be also python3, right?

[int19h]

Yep, I think that should do the trick.

[EimantasBlazevicius]

Good day Gents, risking a dumb question:
How does one change "The "pythonPath" setting in debug configuration"

[bwateratmsft]

Not a dumb question at all! You can add it to the launch configuration, but it will show yellow squiggles:

@int19h I tried setting it (as above), but when I launch it seems like nothing happens. No output in the terminal, output, or debug console tabs. The same happens when I set it to the absolute path of the Python executable in the image. When I run without that argument I get the error this bug is about.

EDIT: I got it to work. However, I'm not really a fan of this solution. Forcing the same Python path for both sides in remote debugging means that either the Python executables have to be in the exact same path on both sides--literally not possible for debugging Linux containers from a Windows machine--or otherwise use "pythonPath": "python", which means the user cannot specify the Python path on the host side to use. I think another argument may be needed to overrule the Python path used on the remote side, in our case to always be python3.

[int19h]

Ohh, I see the problem now! Yes, you're right - the Python extension itself will also look at "pythonPath" to determine which interpreter to use to start the adapter.

But we can't just revert the launcher to the old behavior, either - the underlying logic was incorrect, because it didn't properly handle stubs like pythonw. I think we'll need some kind of flag for the launcher to opt out of the new behavior, or maybe just a way to override the version of Python that it uses, disregarding the debug configuration. In the meantime, I would suggest pinning debugpy to version 1.0.0 - the change was made in 1.1.0.

[bwateratmsft]

@int19h Is there a way to pin the version from the launch config? Or something else?

[int19h]

@bwateratmsft If you're installing debugpy in the container, you can choose the specific version at that point. If you're mapping the debugpy directory from Python extension on the host inside the guest, that wouldn't work, though (but there are other issues with this approach - e.g. you won't get any native code optimizations in the debugger, because of OS/arch mismatch).

[bwateratmsft]

@int19h When can we expect a fix for this? It is blocking Python debugging in Docker unless users manually alter the launch.json to set "pythonPath": "python". Pinning debugpy would be a pretty complicated procedure given that we volume map in what the Python extension bundles--the user would have to download and install debugpy locally, then map that in instead of what the Python extension has. I'm not sure if that will result in compatibility issues. Perhaps the simplest workaround for users (other than setting "pythonPath") would be to roll back the Python extension, but I think that prevents future auto-updating unless they manually re-enable it, which isn't ideal.

[mhlg]

@int19h unless users manually alter the launch.json to set "pythonPath": "python".

Where do we alter "pythonPath": "python" in the launch.json

{ "name": "Docker: Python - General", "type": "docker", "request": "launch", "preLaunchTask": "docker-run: debug", "pythonPath": "python", "python": { "pathMappings": [ { "localRoot": "${workspaceFolder}", "remoteRoot": "/app" } ], "projectType": "general" } }

[bwateratmsft]

Like this:

The yellow warning squiggles there will appear, because it's not in the expected schema for Docker launch configurations, but we pass through most variables when resolving it into a Python launch config. You can safely ignore this warning.

[mhlg]

@bwateratmsft I get an erroror after the image is compiled that says

Invalid Message: "debugLauncherPath" must be str

[bwateratmsft]

@mhlg Ah, it's Mac. Try "pythonPath": "python3" instead of "pythonPath": "python".

[bwateratmsft]

Workaround:
Windows: Add "pythonPath": "python" into .vscode/launch.json, as in the image below
Mac/Linux: Add "pythonPath": "python3" into .vscode/launch.json to the same place

The yellow squiggle warning will appear but it can be ignored. Once the file is closed it will no longer show up in the Problems list.

[int19h]

Note that for this to work, it must be on PATH locally.

@bwateratmsft I'm preparing a change that will give you fine-grained control over which interpreters are used where. It basically splits up "pythonPath" into three:

• "debugAdapterPython" - used by the Python extension to spawn the adapter
• "debugLauncherPython" - used by the adapter to spawn the launcher
• "python" - used by the launcher to spawn the debuggee

These can all be set separately. If any of them are not set, the default value is provided by the extension in the same manner it does for "pythonPath" today (i.e. the active interpreter for the current workspace). For backwards compatibility, "pythonPath" will still be accepted, and will behave the same as setting all three properties at once.

For your scenario in particular, this will allow you to set "python" in your configuration resolver to the appropriate path inside the container, without affecting the other two.

I'll have a PR shortly, and we'll try to get it in ASAP.

[bwateratmsft]

Thanks @int19h! We'll set the value for "python" in our resolved configuration once this change is available. Within containers we always want to use python3, no real exceptions.

[bwateratmsft]

In standup today we decided to release the fix as a servicing release (i.e. 1.8.1). I'll plan on that unless the fix reaches the Python extension sooner than expected.

[bwateratmsft]

A fix for this has now been released in Docker extension 1.8.1.