Skip to content
This repository has been archived by the owner on Aug 2, 2023. It is now read-only.

breakpoint are not hit when debugger is attached to remote target #1093

Closed
dim2iv opened this issue Jan 9, 2019 · 20 comments
Closed

breakpoint are not hit when debugger is attached to remote target #1093

dim2iv opened this issue Jan 9, 2019 · 20 comments
Assignees
Milestone

Comments

@dim2iv
Copy link

dim2iv commented Jan 9, 2019

Environment data

  • PTVSD version: 4.2.0
  • OS and version: Linux, Fedora 29
  • Python version: Python 2.7.15
  • Using VS Code or Visual Studio:
    Version: 1.30.2
    Commit: 61122f88f0bf01e2ac16bdb9e1bc4571755f5bd8
    Date: 2019-01-07T22:49:48.319Z
    Electron: 2.0.12
    Chrome: 61.0.3163.100
    Node.js: 8.9.3
    V8: 6.1.534.41
    OS: Linux x64 4.19.13-300.fc29.x86_64

Target machine:

  • OS and version: Linux, Ubuntu 18.04LTS
  • PTVSD version: 4.2.0
  • Python 2.7.15rc1

Actual behavior

I try to debug remotely python script with vscode. In my launch.json i have proper configuration including pathMappings.

I have try following cases:

  1. Run script on target machine like:
    python -m ptvsd --single-session --host 10.3.72.215 --port 3000 --wait --multiprocess script-debug
    On vscode i put break point and attach to target. As result script-debug is executed without hitting break point and on target machine i see message:
    pydev debugger: warning: trying to add breakpoint to file that does not exist: "settings from localRoot /script-debug"
    Some times i can manage to start debugging - i.e. debugger stops at BP, but this is less than 1% of all attempts.

  2. Run script on target but modified:
    _script-debug_

import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# # Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()

Same behavior like p.1

  1. Run script on target but more modified:
    _script-debug_
import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# # Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()
ptvsd.break_into_debugger()

In this case 100% of attempts as successful if i remove/disable all BP from vscode and add/enable them once debugger is attached.

If i start debugger with breakpoints, that are set before attachment to remote target, again see message on remote site:
pydev debugger: warning: trying to add breakpoint to file that does not exist: "settings from localRoot /script-debug"
So each time before debug session i have to disable BP, attach to target and re-enable them in order to use debugger.

Looks like some kind of race condition when debugger is attached and pathMappings, and BP are set to remote side.

Expected behavior

To be possible to run target like 'python -m ptvsd --single-session --host 10.3.72.215 --port 3000 --wait --multiprocess script-debug" and not to be forced to use ptvsd.break_into_debugger() in order to set BP

Already configured BP to work without needs to be re-enabled.

@karthiknadig
Copy link
Member

@dim2iv Can you share the launch.json configuration you are using for this?

@dim2iv
Copy link
Author

dim2iv commented Jan 9, 2019

@karthiknadig following is content of lauch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach (nfvt-up)",
            "type": "python",
            "request": "attach",
            "port": 3000,
            "host": "10.3.72.215",
            "pathMappings": [
                
                { 
                    "localRoot": "${workspaceFolder}/nfvtime-upgrade/orig/", 
                    "remoteRoot": "/opt/bin/" 
                }
            ],
            "logToFile": false,
            "redirectOutput": true,
            "subProcess": true
        },
        {
            "name": "Python: Current File (Integrated Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal"
        },
        {
            "name": "Python: Module",
            "type": "python",
            "request": "launch",
            "module": "enter-your-module-name-here",
            "console": "integratedTerminal"
        },
        {
            "name": "Python: Django",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/manage.py",
            "console": "integratedTerminal",
            "args": [
                "runserver",
                "--noreload",
                "--nothreading"
            ],
            "django": true
        },
        {
            "name": "Python: Flask",
            "type": "python",
            "request": "launch",
            "module": "flask",
            "env": {
                "FLASK_APP": "app.py"
            },
            "args": [
                "run",
                "--no-debugger",
                "--no-reload"
            ],
            "jinja": true
        },
        {
            "name": "Python: Current File (External Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        }
    ]
}

@karthiknadig
Copy link
Member

For clarification, the full path for script-debug on remote is /opt/bin/script-debug?

@dim2iv
Copy link
Author

dim2iv commented Jan 10, 2019

Yes. This path on remote (target) machine.

@karthiknadig
Copy link
Member

@dim2iv I tried this, it seems to be working. Can you share your project? This might be specific to the project.

@dim2iv
Copy link
Author

dim2iv commented Jan 11, 2019

@karthiknadig here is a file. Is it possible reason to be that some of imported modules do not exists on host (locals) system, but exists on target ( remote ) system ?
source.zip

@karthiknadig
Copy link
Member

@dim2iv if the files are missing locally, you get a message box that pops up on VS Code that says 'File not found'.

Unfortunately, I am not able to repro this at all (with the source file you provided). Can you do one more thing, set "logToFile": true, and attach the log file here? You may want to remove any personal info from the log file. The log file should be in C:\Users\<username>\.vscode\extensions\ms-python.python-2018.12.1\debug<pid>.log.

@dim2iv
Copy link
Author

dim2iv commented Jan 11, 2019

@karthiknadig with missing files i mean, that some modules imported do not exists on host system. They are related to (like apt_pkg). They are specific for the Ubuntu (target system), while on development host they are not installed at all. Development host is Fedora.
I will will run with enabled logging and will try to collect and send files for good and bad cases.

@dim2iv
Copy link
Author

dim2iv commented Jan 14, 2019

@karthiknadig Here You can find logs for:
Bad case 1 - script is started on target like this 'python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up'
Bad case 2 - script is modified and code is added at start of file

_#! /usr/bin/env python
import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()_

In both cases bp is set to line 1300
In both cases debugger just catch exception on sys.exit()
Do You need log in case where ptvsd.break_into_debugger() is called ?
bad_case2.zip
bad-case1.zip

@karthiknadig
Copy link
Member

Can you try the following and see if that helps?

  1. Try moving the location on the remote machine to something other than /opt/bin.
  2. Try setting "debugStdLib": true.
    I suspect the issue might be that the debugger is ignoring the break-point because it thinks the break-point is in a standard library and skips it (it should not).

If either of the above steps did not help then, can you give me the location of site-packages and python installs on the remote machine? Run the code below using the same python you use to launch your remote script

import sys
import site
print('SysPath:' + str(sys.path))
print('Prefix: ' + sys.prefix)
if hasattr(sys, 'base_prefix'):
    print('BasePrefix: ' + sys.base_prefix)
if hasattr(sys, 'real_prefix'):
    print('RealPrefix: ' + sys.real_prefix)
if hasattr(sys, 'exec_prefix'):
    print('ExecPrefix: ' + sys.exec_prefix)

if hasattr(site, 'getusersitepackages'):
    print('User pkg: ' + str(site.getusersitepackages()))
if hasattr(site, 'getsitepackages'):
    print('Site pkg: ' + str(site.getsitepackages()))

Let me know how this goes.

@dim2iv
Copy link
Author

dim2iv commented Jan 15, 2019

@karthiknadig i notice something very interesting.
I have execute debug session using p.1 and p.2. I have moved my script in /root directory and start it like python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up
Result - debugger hit planted BP at line 1300. All is ok and can use the debugger.
After that i decide to run another one session. Note that i do not change anything in code of debugged script or in settings.
Result - debugger do not hit planted BP ?!?!?
After that i restart VSCode and repeat again. All was same like before - first run debug session is ok, all following - do not stops at BP. Please see attached debug log files. File with less pid is good one. And i notice by simple compare that in good case there is extra rows in log. Please take a look at line 71. The difference starts from here.
May be key point is restart of VSCode, where some cached information is removed ?

Here is also output for site-packeges:

SysPath:['/root', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
Prefix: /usr
ExecPrefix: /usr
User pkg: /root/.local/lib/python2.7/site-packages
Site pkg: ['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

Sorry for the long post but since we have big time shift i think it can help You if all information is available.
case-first-second.zip

@karthiknadig
Copy link
Member

Thanks you for the logs. It looks like the breakpoint event is not getting triggered in the second case. Can you run the debugger with following env variable on the remote machine?
PTVSD_DEBUG=True

You should see some output like this. Please capture that output for the good and bad case.

sending output
output sent
<- 501 (CMD_VERSION)            [1000000000]: |1.1      WINDOWS ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     C:\GIT\pyscratch3\scratch1.py   3       None
None    None    None    None    ALL|
<- 501 (CMD_VERSION)            [1000000002]: |1.1      WINDOWS ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException    0       1       1|
<- 200 (CMD_REDIRECT_OUTPUT)    [1000000004]: |STDOUT   STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000006]: |{"skip_suspend_on_breakpoint_exception": ["BaseException"], "skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true}|
<- 101 (CMD_RUN)                [1000000007]: ||
ready to debug
<- 102 (CMD_LIST_THREADS)       [1000000008]: ||
-> 502 (CMD_RETURN)             [1000000008]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
-> 103 (CMD_THREAD_CREATE)      [         2]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
-> 105 (CMD_THREAD_SUSPEND)     [         4]: |<xml><thread id="pid_16412_id_2374191782712" stop_reason="111" message="" suspend_type="trace"><frame id="2374208578632" name="&lt;module&gt;" file="C%3A%5CGIT%5Cpyscratch3%5Cscratch1.py" line="3"></frame></thread></xml>\n|
-> 157 (CMD_THREAD_SUSPEND_SINGLE_NOTIFICATION) [         6]: |{"thread_id": "pid_16412_id_2374191782712", "stop_reason": 111}\n|
<- 102 (CMD_LIST_THREADS)       [1000000009]: ||
-> 502 (CMD_RETURN)             [1000000009]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
<- 152 (CMD_GET_THREAD_STACK)   [1000000010]: |pid_16412_id_2374191782712|
.
.
.

@dim2iv
Copy link
Author

dim2iv commented Jan 17, 2019

@karthiknadig You can find logs attached. As i see on bad case ptvsd complains that file is unknown. Looks like mappings are not set second ( and following ) run(s).
good.zip
bad.zip

@karthiknadig
Copy link
Member

karthiknadig commented Jan 17, 2019

@fabioz have you seen this issue. Looks like path mapping, fails often, in this scenario.

good case:

<- 501 (CMD_VERSION)            [1000000000]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up        1300    None    None    None    None    None       ALL|
<- 501 (CMD_VERSION)            [1000000002]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|

bad case:

<- 501 (CMD_VERSION)            [1000000000]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up        1300    None    None    None    None    None       ALL|
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up (will have no effect)
<- 501 (CMD_VERSION)            [1000000002]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|

I see a success response from DAP logs for both cases. Any idea where this might be breaking?

@fabioz
Copy link
Contributor

fabioz commented Jan 17, 2019

Unfortunately the log seems to be incomplete as there's no info on the file translation being set -- and it really seems like the translation is set on one case but not in the other...

@dim2iv, can you do the following:

  1. Set an environment variable named DEBUG_PYDEVD_PATHS_TRANSLATION with a value of 1.

  2. Open the file: pydevd_file_utils.py (to get its location you can create a program with the contents below and run it):

import pydevd_file_utils
print(pydevd_file_utils.__file__)

Then put a sys.stderr.write('setup_client_server_paths: %s\n' % (paths,)) as the first line in the method setup_client_server_paths .

  1. Run that and provide the output on the good/bad case.

@dim2iv
Copy link
Author

dim2iv commented Jan 17, 2019

@fabioz i do not have installed pydevd installed on host or on target machine.
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pydevd_file_utils
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pydevd_file_utils

In VScode is not mentioned that i have to install it in order to able to debug.

@fabioz
Copy link
Contributor

fabioz commented Jan 17, 2019

@dim2iv ptvsd bundles pydevd, so, I thought it'd be there... try to do:

import ptvsd
print(ptvsd.__file__)

then, the location of pydevd_file_utils.py should be under the ptvsd folder /_vendored/pydevd/pydevd_file_utils.py.

@dim2iv
Copy link
Author

dim2iv commented Jan 18, 2019

Hi @fabioz and @karthiknadig,
Requested logs are attached. Application is started on target like:
PTVSD_DEBUG=True DEBUG_PYDEVD_PATHS_TRANSLATION=1 python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up

I think there is a race condition (look at bold)
Good

<- 501 (CMD_VERSION) [1000000000]: |1.1 UNIX ID|
-> 501 (CMD_VERSION) [1000000000]: |@@BUILD_NUMBER@@\n|
setup_client_server_paths: [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')]
<- 111 (CMD_SET_BREAK) [1000000001]: |1 python-line /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up 1300 None None None None
None ALL|
pydev debugger: replacing to server: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up
pydev debugger: sent to server: /root/nfvt-up
<- 501 (CMD_VERSION) [1000000002]: |1.1 UNIX ID|
-> 501 (CMD_VERSION) [1000000002]: |@@BUILD_NUMBER@@\n|
<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException 0 1 0|
<- 200 (CMD_REDIRECT_OUTPUT) [1000000004]: |STDOUT STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000005]: |{"skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true, "skip_suspend_on_breakpoint_exception": ["BaseException"]}|
<- 101 (CMD_RUN) [1000000006]: ||

Bad

<- 501 (CMD_VERSION) [1000000000]: |1.1 UNIX ID|
<- 111 (CMD_SET_BREAK) [1000000001]: |1 python-line /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up 1300 None None None None None ALL|
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up (will have no effect)
<- 501 (CMD_VERSION) [1000000002]: |1.1 UNIX ID|
-> 501 (CMD_VERSION) [1000000000]: |@@BUILD_NUMBER@@\n|
-> 501 (CMD_VERSION) [1000000002]: |@@BUILD_NUMBER@@\n|
setup_client_server_paths: [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')]

<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException 0 1 0|
<- 200 (CMD_REDIRECT_OUTPUT) [1000000004]: |STDOUT STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000005]: |{"skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true, "skip_suspend_on_breakpoint_exception": ["BaseException"]}|
<- 101 (CMD_RUN) [1000000006]: ||
ready to debug

bad-ext.zip
good-ext.zip

I notice following when DEBUG_PYDEVD_PATHS_TRANSLATION is set.
pydev debugger: replacing to client: /root/nfvt-up
pydev debugger: sent to client: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up
pydev debugger: to client: unable to find matching prefix for: /usr/local/lib/python2.7/dist-packages/ptvsd/_local.py in ['/root']
pydev debugger: to client: unable to find matching prefix for: /usr/local/lib/python2.7/dist-packages/ptvsd/main.py in ['/root']
pydev debugger: to client: unable to find matching prefix for: /usr/lib/python2.7/runpy.py in ['/root']

@fabioz
Copy link
Contributor

fabioz commented Jan 18, 2019

@karthiknadig it really seems like ptvsd is sending breakpoints before setting up the translation in this case (which wasn't expected in pydevd)... now, there are 2 possible fixes:

  1. ptvsd makes sure that the paths are setup before the breakpoints.
  2. pydevd itself can update the breakpoints if the translations are sent after breakpoints are set.

If the requests are expected to be sent in order by the DAP, then 1 may be reasonable and if it's out of order, 2 must be done (although 2 must be done if we expect the user to be able to change things programmatically later on).

@dim2iv I think a workaround for you for now is updating the method setup_client_server_paths (in pydevd_file_utils.py:

Put paths = [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')] as the first line of the method until this is actually fixed (this should work because the initial translation is called when that module is imported).

@dim2iv
Copy link
Author

dim2iv commented Jan 18, 2019

@fabioz proposed workaround is too hard coded, since will work just for one script in project. I able to debug for now and will wait for real fix.
@karthiknadig is this is considered as a problem ?

int19h added a commit to int19h/ptvsd that referenced this issue Jan 31, 2019
…to remote target

Do not allow 'setBreakpoints' request until either 'launch' or 'attach' request is received.

Wait for 'launch' or 'attach' request to process path mappings before handling 'setBreakpoints'.
int19h added a commit to int19h/ptvsd that referenced this issue Feb 1, 2019
…to remote target

Do not allow 'setBreakpoints' request until either 'launch' or 'attach' request is received.

Wait for 'launch' or 'attach' request to process path mappings before handling 'setBreakpoints'.
@int19h int19h closed this as completed in 07dc5e4 Feb 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants