Debug multiprocess leads to AttributeError Can't get attribute on <module '__main__

[GoranDubajic]

Environment data

• VS Code version: Version: 1.42.1 (user setup)
  Commit: c47d83b293181d9be64f27ff093689e8e7aed054
  Date: 2020-02-11T14:45:59.656Z
  Electron: 6.1.6
  Chrome: 76.0.3809.146
  Node.js: 12.4.0
  V8: 7.6.303.31-electron.0
  OS: Windows_NT x64 10.0.18363
• Extension version (available under the Extensions sidebar): 2020.2.64397
• OS and version: Windows_NT x64 10.0.18363
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.1, it reproduces for other versions as well.
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: N/A
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version vscode-python#3977): true
• Value of the python.languageServer setting: Microsoft

Expected behaviour

Script should debug without any exception.

Actual behaviour

Exception at line 14 for the script bellow.

Exception has occurred: AttributeError
Can't get attribute 'Foo' on <module '__main__' from 'c:\\Users\\gorand\\.vscode\\extensions\\ms-python.python-2020.2.64397\\pythonFiles\\lib\\python\\new_ptvsd\\no_wheels\\ptvsd\\__main__.py'>
  File "E:\git\VSOGD\Misc\Python\mp_queue_example.py", line 14, in <module>
    print(q.get().value)    # prints 1

Steps to reproduce:

Debug this script.

from multiprocessing import Process, Queue

class Foo:
    def __init__(self, value):
        self.value = value

def f(q):
    q.put(Foo(1))

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get().value)    # prints 1
    p.join()

[fabioz]

I don't have exactly the same error with the latest version, but I do have an error:

The launch is:

        {
            "name": "Python: Current File (Integrated Terminal)",
            "type": "python",
            "request": "launch",
            "program": "X:/ptvsd_workspace/ptvsd_tests/foo/bar.py",
            "console": "integratedTerminal",
            "cwd": "X:/ptvsd_workspace/ptvsd_tests",
            "debugAdapterPath": "D:\\x\\ptvsd_workspace\\ptvsd\\src\\debugpy\\adapter",
            "subProcess": true,
            "redirectOutput": false,
            "logToFile": true,
        },

The logs are:
debugpy.pydevd.4404.log
debugpy.pydevd.17868.log
debugpy.server-4404.log
debugger.vscode_db4faac6-be95-4217-852e-8bdb13718f03.log
debugpy.adapter-5760.log
debugpy.launcher-9044.log

[fabioz]

I believe this is because in a request debugpy seems to be passing subProcessId and not processId (see debugpyAttach in debugger.vscode_db4faac6-be95-4217-852e-8bdb13718f03.log) @karthiknadig @int19h can you give more info here?

-- Note: using plain pydevd this seems to work and I believe this is some issue in the vscode-python/debugpy integration -- note that I'm using the latest nightly build for vscode-python here -- actual version: ms-python.python-2020.4.67731-dev.

[int19h]

@fabioz
"subProcessId" is correct. What's different with debugpy master is that it's passing the host/port info inside "connect" now, rather than the top-level host/port. We have the corresponding change in the extension, it just didn't get into a build yet.

For now, you either need to use microsoft/debugpy@2d013e5 to match the public vscode-python builds, or else build the extension itself from master.

[int19h]

And yes, this is likely to be an issue with sys.argv handling when debugpy is doing command line parsing. I'll investigate.

[fabioz]

For now, you either need to use microsoft/debugpy@2d013e5 to match the public vscode-python builds, or else build the extension itself from master.

Thanks, that does the trick for me.

[int19h]

Turns out this is a quirk of multiprocessing module - it's sensitive to the exact point where it gets imported.

When it spawns child processes, the main script is run as __mp_main__, to avoid running the part that's inside if __name__ == "main": .... This means that in the child process, Foo's qualified name is actually __mp_main__.Foo, and it's pickled as such to marshal it back to the parent process. To make this work in the parent when unpickling, multiprocessing does this:

if '__main__' in sys.modules:
    sys.modules['__mp_main__'] = sys.modules['__main__']

The problem is that it's done once in __init__.py - thus, if __main__ changes later, multiprocessing will continue to use the original main module as it was at the point where multiprocessing was imported.

This just happens to be the case with debugpy - we import multiprocessing early to do some logging, before using runpy to start the script. And runpy updates sys.modules["__main__"], but that doesn't affect __mp_main__.

Technically I think this is rather a bug in runpy, but we can work around this by not importing multiprocessing early - its sole use is to determine CPU count for the log, but I don't recall us ever using this information in any investigations.

[int19h]

@fabioz, it looks like pydevd itself indirectly imports multiprocessing on Python 3.6 and below. I added a traceback to multiprocessing/__init__.py, and got this trying to debug an empty script:

  File "C:\Python\3.6-64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python\3.6-64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\git\debugpy\src\debugpy\__main__.py", line 43, in <module>   
    from debugpy.server import cli
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\git\debugpy\src\debugpy/..\debugpy\server\__init__.py", line 9, in <module>
    import debugpy._vendored.force_pydevd  # noqa
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\git\debugpy\src\debugpy/..\debugpy\_vendored\force_pydevd.py", line 49, in <module>
    'pydevd',
  File "C:\git\debugpy\src\debugpy/..\debugpy\_vendored\__init__.py", line 128, in preimport
    import_module(name)
  File "C:\Python\3.6-64\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\git\debugpy\src\debugpy\_vendored\pydevd\pydevd.py", line 57, in <module>
    from pydevd_concurrency_analyser.pydevd_concurrency_logger import ThreadingLogger, AsyncioLogger, send_concurrency_message, cur_time
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\git\debugpy\src\debugpy\_vendored\pydevd\pydevd_concurrency_analyser\pydevd_concurrency_logger.py", line 33, in <module>
    import asyncio  # @UnresolvedImport
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Python\3.6-64\lib\asyncio\__init__.py", line 21, in <module>
    from .base_events import *
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Python\3.6-64\lib\asyncio\base_events.py", line 17, in <module>
    import concurrent.futures
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Python\3.6-64\lib\concurrent\futures\__init__.py", line 17, in <module>
    from concurrent.futures.process import ProcessPoolExecutor
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Python\3.6-64\lib\concurrent\futures\process.py", line 53, in <module>
    import multiprocessing
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Python\3.6-64\lib\multiprocessing\__init__.py", line 39, in <module>
    import traceback; traceback.print_stack()

The dependency chain is pydevd -> pydevd_concurrency_analyser -> asyncio -> multiprocessing. It doesn't occur on 3.7+, probably because asyncio doesn't import multiprocessing there.

Can this import be deferred? Patching up multiprocessing is possible - but tricky, because it's runpy creating it for user code, and I don't see any easy way to get our code to run after that but before user code runs. We'd basically have to duplicate the entirety of runpy.run_path...

[fabioz]

Thanks for the investigation. I've just fixed the part related to pydevd in microsoft/debugpy#78 (i.e.: asyncio is no longer imported there).