Unable to debug using ptvsd - Mixed mode debugging #1283

sujiths93 · 2019-03-31T01:24:59Z

Environment data

PTVSD version: The version that comes with VS Code Installation.
OS and version: Mac OS Mojave 10.14.3
Python version (& distribution if applicable, e.g. Anaconda): Python 2.7.15
Using VS Code or Visual Studio: VS Code 1.32.3

Actual behavior

I am trying to debug Python/C++ code using these instructions
,however when I launch the Python debugger which walks through the python code having
the statement manager = multiprocessing.Manager() it breaks there and here is the error trace i see :

E-0000.454: Exception escaped from start_client
            
            Traceback (most recent call last):
              File "/Users/sujithshivaprakash/.vscode/extensions/ms-python.python-2019.3.6139/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
                return f(*args, **kwargs)
              File "/Users/sujithshivaprakash/.vscode/extensions/ms-python.python-2019.3.6139/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
                sock, start_session = daemon.start_client((host, port))
              File "/Users/sujithshivaprakash/.vscode/extensions/ms-python.python-2019.3.6139/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
                with self.started():
              File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
                return self.gen.next()
              File "/Users/sujithshivaprakash/.vscode/extensions/ms-python.python-2019.3.6139/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
                self.start()
              File "/Users/sujithshivaprakash/.vscode/extensions/ms-python.python-2019.3.6139/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
                raise RuntimeError('already started')
            RuntimeError: already started
            

Exception occured while processing the plan: Exception occured while processing the plan:  
--- [START] Exception Traceback ---
Traceback (most recent call last):
  File "/Users/sujithshivaprakash/Documents/GitHub/voxsupFrontend2/server/TVAnalytics/fourctv/plan_generation/generate_media_plan.py", line 1833, in run_media_plan_generation
    df_plan_main = optimizer.optimize()
  File "/Users/sujithshivaprakash/Documents/GitHub/voxsupFrontend2/server/TVAnalytics/fourctv/plan_generation/run_optimizers.py", line 364, in optimize
    allow_excess_goal=allow_excess_goal)
  File "/Users/sujithshivaprakash/Documents/GitHub/voxsupFrontend2/server/TVAnalytics/fourctv/plan_generation/run_optimizers.py", line 274, in run_optimizer
    campaign_constraint_priority=self.campaign_constraint_priority,
  File "/Users/sujithshivaprakash/Documents/GitHub/voxsupFrontend2/server/TVAnalytics/fourctv/plan_generation/plan_optimizers.py", line 338, in optimizer_GA_multiprocess
    manager = mp.Manager()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 99, in Manager
    m.start()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 528, in start
    self._address = reader.recv()
EOFError

NOTE : Just running the python code without debugger runs perfectly fine.

Expected behavior

Debugger steps trhough the break-points without crashing.

Steps to reproduce:

Not sure about this.

After having going over articles for sometime now, I dont understand what s going wrong.
Any help appreaciated.
Thanks

The text was updated successfully, but these errors were encountered:

karthiknadig · 2019-04-01T17:40:40Z

@sujiths93 The recommended solution for mixed mode debugging is to use Visual Studio 2017/2019. See here:
https://docs.microsoft.com/en-us/visualstudio/python/debugging-mixed-mode-c-cpp-python-in-visual-studio?view=vs-2017

/cc @qubitron

sujiths93 · 2019-04-01T17:56:12Z

Visual studio with python support is unavailable for Mac.

int19h · 2019-04-01T20:02:56Z

Judging by "already started" in the output and use of multiprocessing, I think it's not really a mixed-mode debugging issue, but rather another manifestation of #943.

@sujiths93, can you please try adding this to your code, making sure that it runs before you use anything else from multiprocessing:

import multiprocessing
multiprocessing.set_start_method('spawn', True)

karthiknadig · 2019-04-01T20:05:45Z

It is pyhton 2.7, that does not support the set_start_method.

aerinkim · 2019-06-12T05:13:00Z

@int19h
I added multiprocessing.set_start_method('spawn', True) however, getting this error.

  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/aerin/anaconda3/envs/sqlova/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'get_loader_wikisql.<locals>.<lambda>'
Terminated

int19h · 2019-06-12T05:37:21Z

@aerinkim This is a general issue with multiprocessing - since separate processes don't share memory space, Python has to pickle and unpickle any data that you pass back and forth.

In this case, it sounds like you tried passing a lambda across processes? The usual way to handle this is by passing a named global function. If the lambda body uses any variables from the local scope, you'll have to pass them as explicit arguments, and add them to the call on the other side.

omartin2010 · 2019-08-03T01:20:06Z

I had the same issue using the Remote extension, trying to debug a PyTorch training (which uses multiprocessing for the data loaders) and adding these lines :

import multiprocessing
multiprocessing.set_start_method('spawn', True)

solved it. Thanks!! :) Looking forward for the fix. I'm running VSCode on Win10 and I remote to a notebook VM in Azure.

Version: 1.37.0-insider (user setup)
Commit: 310cb67b0e5df5b0644f42df93ada186130dcd1b
Date: 2019-08-02T11:11:48.598Z
Electron: 4.2.7
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 10.0.18362

And I can repro anytime, feel free to ping me on Teams if that helps to see the repro.

gramster · 2019-09-03T23:39:46Z

This problem should go away once we support fork.

juskuz · 2020-01-14T09:00:25Z

@omartin2010 where did you put these lines (which file)?
I'm trying to debug my code in VSC running in Win10 using WSL.

After adding these lines to my model.py file (there train() is called) I get similar error as aerinkim...

File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp)
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'GeneratorEnqueuer.start.<locals>.data_generator_task'

omartin2010 · 2020-01-14T15:44:51Z

if I recall (it's been a few months) it in the first file that was started. Perhaps it needs to be where you're spawning your threads... are you running Python 3.6 or 3.7? and a recent version of ptvsd?

int19h · 2020-01-14T16:53:27Z

@juskuz, the error that you get is because "spawn" has more constraints than "fork" - and, in particular, requires any data shared between the processes to be pickleable.

At this point, I'd recommend trying out the new debugger with support for fork. You can opt into it by adding this to your settings.json:

"python.experiments.optInto": [
    "DebugAdapterFactory - experiment",
    "PtvsdWheels37 - experiment"
]

then remove set_start_method()

juskuz · 2020-01-15T14:20:09Z

@omartin2010 Thanks for reply.

are you running Python 3.6 or 3.7? and a recent version of ptvsd?

Python 3.6.9
ptvsd 5.0.0a11

@int19h, the new debugger is ptvsd == 5.0.0a11?

omartin2010 · 2020-01-15T15:17:42Z

It's been too long, I don't recall what I've used. Likely Python 3.6.x.

int19h · 2020-01-15T19:31:30Z

@juskuz Any minor release of ptvsd 5 should work, not necessarily a11.

juskuz · 2020-01-16T09:43:08Z

ok, I added the

"python.experiments.optInto": [
    "DebugAdapterFactory - experiment",
    "PtvsdWheels37 - experiment"
]

to my to settings.json and removed set_start_method().

Now I'm getting 'RuntimeError: already started' again.

and at the end of error messages I get

  File "/home/wsl-user/.local/lib/python3.6/site-packages/keras/engine/training.py", line 2065, in fit_generator
    generator_output = next(output_generator)
StopIteration
Terminated

I used ptvsd 5.0.0a11, it's the newest available version.

fabioz · 2020-01-16T11:38:40Z

@juskuz Have you restarted after doing that change to the settings?

Can you double check that you're using the ptvsd 5 version?

-- You can use the code below when running under the debugger to get the version.

import ptvsd
print(ptvsd.__version__)
print(ptvsd)

karthiknadig · 2020-01-16T20:44:30Z

@juskuz You will need the insiders version of the python extension, you can download it here https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix.

sujiths93 changed the title ~~Unable to debug using ptvsd.~~ Unable to debug using ptvsd. EOF Error Mar 31, 2019

karthiknadig changed the title ~~Unable to debug using ptvsd. EOF Error~~ Unable to debug using ptvsd - Mixed mode debugging Apr 1, 2019

karthiknadig added Enhancement Mixed-mode-debugging labels Apr 1, 2019

karthiknadig removed the Enhancement label Apr 1, 2019

karthiknadig mentioned this issue Jun 27, 2019

Runtime Error: Already started in multiprocess #1443

Closed

int19h mentioned this issue Jul 23, 2019

[vscode]: TypeError: cannot serialize '_io.TextIOWrapper' (pydevd_bundle.pydevd_io.IORedirector) #1607

Closed

gramster closed this as completed Sep 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to debug using ptvsd - Mixed mode debugging #1283

Unable to debug using ptvsd - Mixed mode debugging #1283

sujiths93 commented Mar 31, 2019

karthiknadig commented Apr 1, 2019 •

edited

Loading

sujiths93 commented Apr 1, 2019

int19h commented Apr 1, 2019

karthiknadig commented Apr 1, 2019

aerinkim commented Jun 12, 2019

int19h commented Jun 12, 2019

omartin2010 commented Aug 3, 2019 •

edited

Loading

gramster commented Sep 3, 2019

juskuz commented Jan 14, 2020

omartin2010 commented Jan 14, 2020

int19h commented Jan 14, 2020

juskuz commented Jan 15, 2020

omartin2010 commented Jan 15, 2020

int19h commented Jan 15, 2020

juskuz commented Jan 16, 2020

fabioz commented Jan 16, 2020

karthiknadig commented Jan 16, 2020

Unable to debug using ptvsd - Mixed mode debugging #1283

Unable to debug using ptvsd - Mixed mode debugging #1283

Comments

sujiths93 commented Mar 31, 2019

Environment data

Actual behavior

Expected behavior

Steps to reproduce:

karthiknadig commented Apr 1, 2019 • edited Loading

sujiths93 commented Apr 1, 2019

int19h commented Apr 1, 2019

karthiknadig commented Apr 1, 2019

aerinkim commented Jun 12, 2019

int19h commented Jun 12, 2019

omartin2010 commented Aug 3, 2019 • edited Loading

gramster commented Sep 3, 2019

juskuz commented Jan 14, 2020

omartin2010 commented Jan 14, 2020

int19h commented Jan 14, 2020

juskuz commented Jan 15, 2020

omartin2010 commented Jan 15, 2020

int19h commented Jan 15, 2020

juskuz commented Jan 16, 2020

fabioz commented Jan 16, 2020

karthiknadig commented Jan 16, 2020

karthiknadig commented Apr 1, 2019 •

edited

Loading

omartin2010 commented Aug 3, 2019 •

edited

Loading