19 Beta 1 Python debugger broken

[fbacher]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

The pydevd debugger worked fine with 19.0a1-Matrix. But by 19.0a2-Matrix the Python debugger never returns from the call to initialize it (pydevd.settrace()). I confirmed that it still fails with 19.0a3-Matrix and Beta 1.

I have created a build environment for Kodi and will work to narrow down when the failure first occurred.

Expected Behavior

Here is a clear and concise description of what was expected to happen:

On the off-chance you don't know how to use pydevd for debugging addons, it is basically:
Find script.module.pydevd (on github, not a repository). Install the addon.
Install the latest pydev plugin for your IDE.
To get the pydev versions for IDE and Kodi plugin to match, you will have to copy the appropriate code from your IDE and overrwrite the script.module.pydevd/lib code.
In your addon's main call:
pydevd.settrace('localhost', stdoutToServer=True,
stderrToServer=True, suspend=False,
wait_for_ready_to_run=True)
Be sure to turn on the debugger in your IDE

Actual Behavior

Instead of returning from settrace, above, it blocks forever.

I also notice that tracemalloc() also locks up. I have never used it before, but I suspect the problems are related.

Possible Fix

To Reproduce

Steps to reproduce the behavior:

Debuglog

The debuglog can be found here:
https://paste.kodi.tv/owetemoxat.kodi
(Nothing exciting). Look for my comment before call to settrace: 'back_end_service trying to attach to debugger'
It should print a message indicating that it returned from call.

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:

• Android

• iOS

• tvOS

• [x ] Linux

• OSX

• Windows

• Windows UWP

• Operating system version/name: Ubuntu 20.04, Python 3.8

• Kodi version: Kodi (19.0-ALPHA2 (18.9.702) Git:20201005-54be31bc5c). Platform: Linux x86 64-bit
  Also Alpha3 & Beta1

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required.
Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[enen92]

@fuzzard maybe you have any idea about this one?

[enen92]

@fbacher can you please provide clear instructions (step by step) on how to reproduce the issue?

[fbacher]

I am doing more investigation. It is proving complicated. I suggest that this should be put on hold until I am done. The debugger DID work with my first sandbox alpha1 build, and I opened this bug once I confirmed this. However, further testing is giving inconsistent results.

Development of my plugin is on-going and I regularly use the debugger. Development was paused for the past month.

I am testing multiple versions of pydevd as well as hand-built kodi versions. Also testing different versions of my plugin. On rare occasions I can get my current plugin build to work with debugger on the Beta-1 build. Similar results running with running on hand-built Alpha-1 build. When it does work it tends to work on several multiple runs.

I can reliably debug if I move the pydevd.settrace() when I put it in a separate thread at startup instead of at beginning of main.

[enen92]

Have you tried https://kodi.wiki/view/HOW-TO:Debug_Python_Scripts_with_Web-PDB?

[fbacher]

Web-PDB worked great! On the surface it looks painful to use on a large app, but I'll check into it.

I'll continue to try to figure out what is going wrong with pydevd.

[fbacher]

Have test case plugin which I attached below, but also put on dropbox. I'm happy to place it elsewhere: https://www.dropbox.com/sh/je8t83k6ltdf6os/AABVZlHJVWhPczz974fqGBJla?dl=0

script.video.pydevtest.zip

New log at https://paste.kodi.tv/lovulewaki.kodi

See pydevdtest.py for instructions on how to reproduce.
Basically, you will need an IDE (Eclipse,LiClipse or PyCharm) with pydevd installed. Also pydevd plugin installed on Kodi.
The test works for Alpha1 but fails for Beta1.
The work around for Beta1 is to run pydevd.settrace() in a separate thread and have main thread loop on xbmc.Monitor.waitForAbort(0.1).
You will have to enable the debugger in the IDE

I noticed similar behavior (but not with pydevd) back in Kodi 18 days. I'll work to reproduce.

[fbacher]

The workaround that worked in my testcase above does not work for my real plugin. While I am able to get the debugger to stop at a breakpoint set the line after the pydevd.settrace() in my code, single stepping several lines leads to the debugger and plugin locking up. I have spent a lot of time trying different things to try to work around. I tried using web-pdb to debug the pydevd problem, but they don't play well together so at least for now I have abandoned that approach.

[fbacher]

Bug occurs with first merge after Alpha2 was created.

Kodi (19.0-ALPHA2 (18.9.702) Git:20201006-7b7c1db4ba). Platform: Linux x86 64-bit

Commit summary:
7b7c1db ("Merge pull request #18357 from fuzzard/python_cleanup", 2020-10-06)

[fuzzard]

Can you provide archives (or better yet repos with all changes necessary to run) for whatever you are using.

[fbacher]

I modified versioncheck to add debugging. You don't actually have to modify addon.xml, if you are sure that pydevd plugin is installed. Make sure that pydev in pydevd/lib is same as what your ide is using.
This log properly passes (fgrep settrace *.log returns: ERROR : versioncheck service settrace worked)
You can also stop at breakpoints after settrace and debug as normal. Note that you can't exit ide without killing kodi

kodi.versioncheck.alpha1.log

This log uses my build with merge 7b7c1db in it. Note that fgrep settrace returns nothing. Further, you can't stop at breakpoints, etc.
kodi.versioncheck.alpha2+.log

Modified versioncheck
script.module.pydevd.zip

Pydevd that I am using. It is the basic pydevd + the particular version of pydev that my IDE is using
service.xbmc.versioncheck.zip

[fuzzard]

Try the following diff. Will need testing to insure no side effects.

As there is an interpreter already up and running by this point, it should be fine to use PyInterpreterState_ThreadHead and not worry about creating a new threadstate.

--- a/xbmc/interfaces/python/PythonInvoker.cpp
+++ b/xbmc/interfaces/python/PythonInvoker.cpp
@@ -188,7 +188,7 @@ bool CPythonInvoker::execute(const std::string& script, const std::vector<std::w
       extern PyThreadState* savestate;
       PyEval_RestoreThread(savestate);
 #else
-      PyThreadState* ts = PyThreadState_New(PyInterpreterState_Main());
+      PyThreadState* ts = PyInterpreterState_ThreadHead(PyInterpreterState_Main());
       PyEval_RestoreThread(ts);
 #endif
       l_threadState = Py_NewInterpreter();

[fbacher]

It appears to work with versioncheck, but not with my large plugin in development. Will test a bit with other shipping plugins.

[fbacher]

Definitely better. I am seeing one case where, depending upon timing, the breakpoints in (at least) one thread are ignored even though it is produce debug logging. I will have to experiment a bit to create a simple test case.

[fbacher]

It appears to be as good as Alpha1. Still, there is this odd behavior that I need to look at more. (Been busy).

[fbacher]

I have sat on this too long. The fix here addresses the debugger problem. I will open a separate issue when I can reliably reproduce the other problem that I have encountered.

[fbacher]

What do I need to do to approve the fix so that it gets pulled into the product?

[bigretromike]

@fbacher so pydev is not working with Kodi19 right? That would explain why I can't make it run.

[fbacher]

I haven't spent much time trying to get the debugger to work lately.

It worked sometime in Alpha1 (I can recreate the build)

I still have the Alpha3 patch build that at least partially fixed it.
Over the past several days I have made multiple experiments:

Currently, the debugger is completely broken in 2.19.0+git20210219.1315-f44fdfbf67-0+final-focal.

Eclipse running with pydevd 8.2.0.202102211157 plugin. Copied Eclipse pydevd code over Kodi pydevd lib so they are identical.

Patched Alpha3

Using the "VersionCheck" plugin as a test:

Run debugger in Separate thread = True
suspend = True
wait_for_ready_to_run = True
Works

Run debugger in Separate thread = True
suspend = False
wait_for_ready_to_run = False
works

Run debugger in Separate thread = False
suspend = False
wait_for_ready_to_run = False
Works

Current build (version above)

Run debugger in Separate thread = True
suspend = False
wait_for_ready_to_run = False
Debugger successfully enters, but app hung

Run debugger in Separate thread = False
suspend = False
wait_for_ready_to_run = False
Debugger locks up, does not return from initialization.
App hung

---

Using a complex multi-threaded plugin (unreleased randomtrailers):

I could not make the current release of the debugger work, even after many tricks. My plugin is multi-threaded.

Even with the Alpha3 patch, the debugger behaves a bit oddly in complex multi-threaded applications.
Starting debugger with minimal imports at beginning of main, and in main thread:
settrace( suspend=False, wait_for_ready_to_run=True) locks up Kodi. No threads displayed in Eclipse Debug Server dialog.

• Same result no matter Suspend or wait_for_ready_to_run settings.

Starting debugger near beginning of main, but in main thread:

• pydevd.settracepydevd.settrace(Suspend=True, wait_for_ready_to_run=True) locks up Kodi. No threads displayed in Eclipse Debug Server dialog. xbmc.log after settrace call never prints.

• Same result with suspend=True, wait_for_ready_to_run=True

Starting debugger with minimal imports at beginning of main, but in separate thread:
settrace.suspend=True, wait_for_ready_to_run=True works as expected: automatically stops at statement after call, no explicit breakpoint needed.

Starting with settrace(suspend=True, wait_for_ready_to_run=False, causes Eclipse dialog with "It seems an old version of the PyDev Debugger is being used..." (it is the correct version of pydevd). Expected debugger to stop at statement after settrace. Perhaps both a required? That would be odd.

Starting with settrace(suspend=False, wait_for_ready_to_run=True, appears to work properly.

Starting debugger near beginning of main, but in separate thread:

• pydevd.settrace(suspend=False, wait_for_ready_to_run=False) appears to work.
• pydevd.settrace(suspend=True, wait_for_ready_to_run=False) causes Eclipse dialog with "It seems an old version of the PyDev Debugger is being used..." (it is the correct version of pydevd). Correct behavior is for debugger to stop at statement after settrace.
• pydevd.settrace(Suspend=True, wait_for_ready_to_run=True) does not result in dialog, but only two threads showing as running in Eclipse Debug Server dialog. No variables displayed for current thread. Application is hung. Normally this should result in debugger stopping at statement following call at an internally generated breakpoint.
• pydevd.settrace(suspend=False, wait_for_ready_to_run=True) appears to work properly. You are able to manually set a breakpoint at statement after settrace and debugger stops. Without wait_for_ready_to_run=True, the breakpoint is missed since the debugger is not completely initialized after run.

[fbacher]

@fbacher so pydev is not working with Kodi19 right? That would explain why I can't make it run.

Looks like this was put into master, but not in release. I assume it will go into next update.
I have a Linux build with the patch, if you like.

[bigretromike]

@fbacher so pydev is not working with Kodi19 right? That would explain why I can't make it run.

Looks like this was put into master, but not in release. I assume it will go into next update.
I have a Linux build with the patch, if you like.

The information about this issue was enough for me, I manage to get it working somehow.

[yodidon]

Hello,
since my switch to kodi 19 jre can no longer debug with Eclipse. do you know a solution? thank you

[fbacher]

I need to look into this some more. I twas working in the pr-release, but stopped along the way. I'll try to track down the problem by Christmas.

[yodidon]

I need to look into this some more. I twas working in the pr-release, but stopped along the way. I'll try to track down the problem by Christmas.

thank you for your answer, I will wait to hear from you

[fbacher]

Looks like we are SOL. The fix is in Kodi 20, but not Matrix. Sigh. A one line change made a year ago.

In fairness, I did take a while to verify the fix. And I have been seeing weirdness with debugging for quite a while that I wanted to get to the bottom of and simply ran out of time. But I thought that I had properly indicated that the fix was much better than what was there before. (Since then I can run with the patch fairly happily).

[fbacher]

Note that applying the patch to 19.0-Matrix and 19.1-Matrix works while 19.3-Matrix and 19.2-Matrix does not work. Perhaps related is a failure to determine or set the locale.
locale.setlocale(locale.LC_ALL, 'en_US') blows up in python.

Will verify with release 20, assuming that it is in any shape to test.

[yodidon]

unfortunately I am under kodi 19.3 on linux and I cannot downgrade...

[fbacher]

fyi: I don't believe that locale has anything to do with this

[fbacher]

This fix does work with Kodi 19.3-Matrix. I think that I must not have restarted Eclipse between tests. (The fix is not in 19.3-Matrix, you have to apply fix and build it).