Everything runs slowly after debugger is called

[battaglia01]

Issue Report Checklist

• Searched the issues page for similar reports
• Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• Could not reproduce inside jupyter qtconsole (if console-related)
• Tried basic troubleshooting (if a bug/error)
  • Restarted Spyder
  • Reset preferences with spyder --reset
  • Reinstalled the latest version of Anaconda
  • Tried the other applicable steps from the Troubleshooting Guide
• Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

After calling the debugger and pressing "c", things seem to run slowly in Spyder. Here is a very simple script to show this:

import pdb

x = 0
while True:
    x += 1
    if x % 10000 == 0:
        print(x)
    if x % 1000000 == 0:
        pdb.set_trace()
        # at this point, press "c"

When this script is run, it will count up to 1000000, printing the value of x every 10000 counts, and then call pdb.set_trace() for the first time. This happens fast enough to be almost instaneous on my computer; of course it doesn't take much time for Python to count to 1000000.

Once you are in pdb, you can press "c" to resume execution, and then you will see the counts of 10000 being printed extremely slowly in comparison. On my computer, it's about one or two every second.

This also seems to happen if, for instance, you replace the pdb.set_trace() above with pass and set a debugger breakpoint there instead.

Versions

Versions:

• Spyder 5.4.2
• Python 3.10.9 64-bit
• Qt 5.15.6
• PyQt5 5.15.7
• Darwin 21.6.0 (macOS Monterey, M1 Max)

Dependencies

I don't see any "Dependencies" menu entry on Mac. Where is this??

[ccordoba12]

Hey @battaglia01, thanks for reporting. I was able to reproduce your problem following your instructions.

@impact27, do you know what could be happening in this case?

[impact27]

Well the debugger needs to check if there is a breakpoint at each instruction, and this is a pathological case python does almost no computations and a lot of instructions. optimising the debugger should be possible but it won’t eliminate the problem completely

[impact27]

From what I see, most of the time is spent in _is_in_decorator_internal_and_should_skip (IPython/core/debugger.py:934) (~70% of the dispatch time) so this is an IPython issue.

[impact27]

You can check by executing skip_predicates debuggerskip False in the debugger, it should be much faster.

[battaglia01]

Hi @impact27 some responses

1. This also happens on real programs which don't just pass
2. It doesn't happen in just iPython
3. It does get marginally faster with skip_predicates debuggerskip False but still slow enough to ruin the program

Thank you!

[impact27]

spyder-kernels depends on ipython so any bug there will be here as well. You can use a tool like https://github.com/benfred/py-spy to check what is taking time for your setup. (The command is sudo py-spy record -o profile.svg -- python3 -m spyder_kernels.console and then console > connect to existing kernel > paste the kernel json file name)

But do keep in mind that the debugger will always add overhead and that you will never be as fast as not having a debugger.

[battaglia01]

Not sure what to say; I get you are saying this is an upstream IPython bug but it doesn't happen at all in regular IPython for me...

[impact27]

So there must be something different with your setup. If I:
1 - Create a file test.py containing:

for x in range(1000000):
    if x % 10000 == 0:
        print(x)
print("done")

2 - run in the spyder console %run test.py : It finishes instantaneously
3 - run in the spyder console %run -d test.py followed by b 4 to add a breakpoint on line 4, and c to continue, it takes ~ 4s
4 - open ipython and do the same, I don't see a difference in execution time compared to spyder_kernels

So if your problem is that the spyder kernel is much slower than ipython, then I can't reproduce your bug unfortunately.

[battaglia01]

Ah! You replaced the pdb.set_trace() with the IPython debugger. OK, yes, very interesting.

So to be clear, this doesn't cause any problems for me on ipython

import pdb

x = 0
while True:
    x += 1
    if x % 10000 == 0:
        print(x)
    if x % 1000000 == 0:
        pdb.set_trace()
        # at this point, press "c"

But this does

# run with %run -d test.py, then type b 8 to set breakpoint to "pass" and then c to run
x = 0
while True:
    x += 1
    if x % 10000 == 0:
        print(x)
    if x % 1000000 == 0:
        pass # set breakpoint here
        # at this point, press "c"

So you are right, this does seem to be upstream. I guess the only reason it was happening in Spyder is because Spyder seems to turn pdb.set_trace() calls into ipdb calls.

[impact27]

Ha! The difference is that IPython disables the debugger when there is no breakpoints left, as there is no way of adding a breakpoint after pressiong c. Spyder doesn't because you might add a breakpoint from the editor.

[ccordoba12]

@impact27, what if we add a command to our debugger that doesn't check for breakpoints on every frame? That could be entered by users like @battaglia01 that wouldn't like to rely on that check.

That command could be run using the exclamation magic (!).

[impact27]

So you want a command to exit the debugger completely? We can add a "exit debugger" command which would remove the debugger overhead. But I am not sure I am convinced?

We can also try to optimise the debugger, which would improve the situation a bit

[HendrikHuel]

Hello,

I am on a MAC (Pro M1 MAX) too and face the same problem as @battaglia01 when using spyder's debugger. With the comments in this thread and an old stackoverflow thread (https://stackoverflow.com/questions/53381373/ipython-console-in-spyder-extremely-slow-in-anaconda) I could pin down the problem to ipython 8.x. I testet ipython 8.9 and 8.10 and benched it against 7.34.

Here is what I did. I used a slightly modified code

from datetime import datetime

x = 0
start = datetime.now()
while True:
    x += 1
    if x % 20000 == 0:
        print(x)
    if x % 100000 == 0:
        print("hello")
        break
end = datetime.now()
print(f"Runtime: {end-start}")
print("Set breakpoint here")

Without a breakpoints the runtime of the code withipython 7.34 and 8.x is about 0.01 seconds on my machine. However,
when is set a breakpoint in the last row and debug the code using spyders debugger the runtime with ipython 8.x is 3.6 seconds and with ipython 7.34 only 0.29 seconds.

Here is what I did to set up both environments: The default environment with ipython 8.11 was created via

conda create -n spyder_debugger8 python=3.9
conda activate spyder_debugger8
conda install spyder

And the benchmark env was created by installing ipython 7.34 like so

conda create -n spyder_debugger7 python=3.9
conda activate spyder_debugger7
conda install spyder
pip install ipython==7.34

As a result I have the follwing packages installed

spyder-5.4.2
spyder-kernels-2.4.2
ipython-8.10.0 / 7.34.0
pyqt-5.15.7

Hopefully my small test helps to understand the runtime problem a little bit better. In the meantime, I'll downgrade ipython on my machine as a workaround.

EDIT:
The environment of your application need to run on ipython 7.34 as well, not only your spyder environment. Otherwise the workaround does not work.

[ba05]

I can confirm reverting to ipython 7.33 made debug run much faster.

[ccordoba12]

This was already reported to the IPython team (see issue ipython/ipython#13972), but we haven't received an answer yet. However, we're considering how to solve this problem on our side.

[ccordoba12]

Hey guys, I think the situation will be improved in our next version (5.4.4) because we disabled some IPython functionality that was making our debugger quite slow. So, the situation should be similar to what you experienced with Spyder 5.3.3

However, more improvements will come in Spyder 6 about this (that's why this issue is marked for that version).