Pdb sometimes raises exception when trying to remove a breakpoint defined in a different debugger session

[pppery]

BPO	24160
Nosy	@gvanrossum, @taleinat, @xdegaye, @pppery, @pablogsal, @iritkatriel
PRs	bpo-24160: Fix breakpoints persistence across multiple pdb sessions #21989 bpo-24160: Fix test_pdb refleaks failure #25182

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-04.15:05:35.176>
created_at = <Date 2015-05-10.16:30:16.842>
labels = ['type-bug', 'library', '3.10']
title = 'Pdb sometimes raises exception when trying to remove a breakpoint defined in a different debugger session'
updated_at = <Date 2021-04-04.15:05:35.175>
user = 'https://github.com/pppery'

bugs.python.org fields:

activity = <Date 2021-04-04.15:05:35.175>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-04.15:05:35.176>
closer = 'pablogsal'
components = ['Library (Lib)']
creation = <Date 2015-05-10.16:30:16.842>
creator = 'ppperry'
dependencies = []
files = []
hgrepos = []
issue_num = 24160
keywords = ['patch']
message_count = 12.0
messages = ['242861', '243255', '376044', '380406', '390077', '390079', '390186', '390187', '390188', '390189', '390192', '390193']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'taleinat', 'xdegaye', 'ppperry', 'Tomer Chachamu', 'pablogsal', 'iritkatriel']
pr_nums = ['21989', '25182']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24160'
versions = ['Python 3.10']

[pppery]

>>> import pdb, test3
>>> pdb.run("reload(test3)")
> <string>(1)<module>()
(Pdb) s
--Call--
> c:\documents and settings\perry\desktop\coding_projects\python\test3.py(1)<module>()
-> foo = 7789
(Pdb) b 2
Breakpoint 1 at c:\documents and settings\perry\desktop\coding_projects\python\test3.py:2
(Pdb) c
> c:\documents and settings\perry\desktop\coding_projects\python\test3.py(2)<module>()
-> bar = 7788
(Pdb) c
>>> pdb.run("reload(test3)")
> <string>(1)<module>()
(Pdb) s
--Call--
> c:\documents and settings\perry\desktop\coding_projects\python\test3.py(1)<module>()
-> foo = 7789
(Pdb) b 1
Breakpoint 2 at c:\documents and settings\perry\desktop\coding_projects\python\test3.py:1
(Pdb) cl 1

Traceback (most recent call last):
  File "<pyshell#592>", line 1, in <module>
    pdb.run("reload(test3)")
  File "C:\Python27\lib\pdb.py", line 1238, in run
    Pdb().run(statement, globals, locals)
  File "C:\Python27\lib\bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "test3.py", line 1, in <module>
    foo = 7789
  File "C:\Python27\lib\bdb.py", line 51, in trace_dispatch
    return self.dispatch_call(frame, arg)
  File "C:\Python27\lib\bdb.py", line 80, in dispatch_call
    self.user_call(frame, arg)
  File "C:\Python27\lib\pdb.py", line 148, in user_call
    self.interaction(frame, None)
  File "C:\Python27\lib\pdb.py", line 210, in interaction
    self.cmdloop()
  File "C:\Python27\lib\cmd.py", line 142, in cmdloop
    stop = self.onecmd(line)
  File "C:\Python27\lib\pdb.py", line 279, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "C:\Python27\lib\cmd.py", line 221, in onecmd
    return func(arg)
  File "C:\Python27\lib\pdb.py", line 622, in do_clear
    err = self.clear_bpbynumber(i)
  File "C:\Python27\lib\bdb.py", line 297, in clear_bpbynumber
    self._prune_breaks(bp.file, bp.line)
  File "C:\Python27\lib\bdb.py", line 268, in _prune_breaks
    self.breaks[filename].remove(lineno)
ValueError: list.remove(x): x not in list
Running the same code without first defining a breakpoint (in the second debugger instance) raises KeyError: [path to test3.py] on the same lien
The contents of test3.py are irrelevant, as long as it is at least two lines long and syntactically correct.

[xdegaye]

I can reproduce the problem on python 3.5 with test3.py as:
def foo():
foo = 7789
bar = 7788

$ python
Python 3.5.0a4+ (default:8bac00eadfda, May  6 2015, 17:40:12) 
[GCC 4.9.2 20150304 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb, test3
>>> pdb.run('test3.foo()')
> <string>(1)<module>()
(Pdb) step
--Call--
> /home/xavier/tmp/test3.py(1)foo()
-> def foo():
(Pdb) break 3
Breakpoint 1 at /home/xavier/tmp/test3.py:3
(Pdb) continue
> /home/xavier/tmp/test3.py(3)foo()
-> bar = 7788
(Pdb) continue
>>> pdb.run('test3.foo()')
> <string>(1)<module>()
(Pdb) step
--Call--
> /home/xavier/tmp/test3.py(1)foo()
-> def foo():
(Pdb) break                                   # 'break' lists no breakpoints.
(Pdb) break 2
Breakpoint 2 at /home/xavier/tmp/test3.py:2
(Pdb) break                                   # 'break' lists two breakpoints.
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at /home/xavier/tmp/test3.py:3
        breakpoint already hit 1 time
2   breakpoint   keep yes   at /home/xavier/tmp/test3.py:2
(Pdb) 

On the second debugging session, the first 'break' command lists no
breakpoints, while the second 'break' command lists two breakpoints including
the one set in the first debugging session.

The problem is that the 'breaks' attribute of the Pdb instance is inconsistent
with the 'bplist' and 'bpbynumber' class attributes of the bdb.Breakpoint
class when the second debugging session is started.

[iritkatriel]

I've submitted a patch that I believe fixes this problem. It adds in Bdb's __init__ a call to a function that reads the Breakpoint's 'bplist' and 'bpbynumber' class attributes and populates the new instances' 'breaks' dict.

[taleinat]

Irit, I like the approach of your PR for an immediate fix, that we could consider backporting.

In the long term, ISTM that we should refactor to store the breakpoints with only a single source of truth, and avoid the duplication between Breakpoint.bplist, Breakpoint.bpbynumber and Bdb.breaks.

[gvanrossum]

New changeset ad442a6 by Irit Katriel in branch 'master':
bpo-24160: Fix breakpoints persistence across multiple pdb sessions (GH-21989)
ad442a6

[iritkatriel]

Thanks!

[pablogsal]

Unfortunately PR21989 has breaking the refleaks buildbots. Example:

https://buildbot.python.org/all/#/builders/320/builds/226/steps/5/logs/stdio

To reproduce:

❯ ./python -m test test_pdb -R 3:3
0:00:00 load avg: 1.40 Run tests sequentially
0:00:00 load avg: 1.40 [1/1] test_pdb
beginning 6 repetitions
123456
.test test_pdb failed -- Traceback (most recent call last):
  File "/home/pablogsal/github/cpython/Lib/doctest.py", line 2205, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions
  File "/home/pablogsal/github/cpython/Lib/test/test_pdb.py", line 326, in test_pdb_breakpoints_preserved_across_interactive_sessions

---

File "/home/pablogsal/github/cpython/Lib/test/test_pdb.py", line 330, in test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions
Failed example:
with PdbTestInput([ # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
'import test.test_pdb',
'break test.test_pdb.do_something',
'break test.test_pdb.do_nothing',
'break',
'continue',
]):
pdb.run('print()')
Expected:
> <string>(1)<module>()
(Pdb) import test.test_pdb
(Pdb) break test.test_pdb.do_something
Breakpoint 1 at ...test_pdb.py:...
(Pdb) break test.test_pdb.do_nothing
Breakpoint 2 at ...test_pdb.py:...
(Pdb) break
Num Type Disp Enb Where
1 breakpoint keep yes at ...test_pdb.py:...
2 breakpoint keep yes at ...test_pdb.py:...
(Pdb) continue
Got:
> <string>(1)<module>()->None
(Pdb) import test.test_pdb
(Pdb) break test.test_pdb.do_something
Breakpoint 1 at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:396
(Pdb) break test.test_pdb.do_nothing
Breakpoint 2 at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:393
(Pdb) break
Num Type Disp Enb Where
1 breakpoint keep yes at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:396
2 breakpoint keep yes at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:393
(Pdb) continue
<BLANKLINE>
----------------------------------------------------------------------
File "/home/pablogsal/github/cpython/Lib/test/test_pdb.py", line 350, in test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions
Failed example:
with PdbTestInput([ # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
'break',
'break pdb.find_function',
'break',
'clear 1',
'continue',
]):
pdb.run('print()')
Expected:
> <string>(1)<module>()
(Pdb) break
Num Type Disp Enb Where
2 breakpoint keep yes at ...test_pdb.py:...
3 breakpoint keep yes at ...pdb.py:...
(Pdb) clear 2
Deleted breakpoint 2 at ...test_pdb.py:...
(Pdb) clear 3
Deleted breakpoint 3 at ...pdb.py:...
(Pdb) continue
Got:
> <string>(1)<module>()->None
(Pdb) break
Num Type Disp Enb Where
2 breakpoint keep yes at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:393
3 breakpoint keep yes at /home/pablogsal/github/cpython/Lib/pdb.py:94
(Pdb) clear 2
Deleted breakpoint 2 at /home/pablogsal/github/cpython/Lib/test/test_pdb.py:393
(Pdb) clear 3
Deleted breakpoint 3 at /home/pablogsal/github/cpython/Lib/pdb.py:94
(Pdb) continue
<BLANKLINE>

test_pdb failed

== Tests result: FAILURE ==

1 test failed:
test_pdb

Total duration: 2.8 sec
Tests result: FAILURE

[pablogsal]

Per our buildbot policy (https://discuss.python.org/t/policy-to-revert-commits-on-buildbot-failure/404) we will need to revert this in 24 hours if is not fixed to avoid masking future errors.

[iritkatriel]

Thanks, I'm looking.

[iritkatriel]

With the patch:

PS C:\Users\User\src\cpython-dev> ./python -m test test_pdb -R 3:3
Running Debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/1] test_pdb
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 34.2 sec
Tests result: SUCCESS

[pablogsal]

New changeset aadd4e1 by Irit Katriel in branch 'master':
bpo-24160: Fix test_pdb refleaks failure (GH-25182)
aadd4e1

[pablogsal]

PR 25182 fixes the issue, so I am closing this again.

Thanks for the quick fix, Irit!