Enhancements to gdb 7 debugging hooks

[davidmalcolm]

BPO	12605
Nosy	@loewis, @warsaw, @birkenfeld, @gpshead, @pitrou, @vstinner, @benjaminp, @davidmalcolm
Files	more-frames-in-gdb-hooks-py3k.patch: py3k version of the patch more-frames-in-gdb-hooks-2.7.patch: 2.7 version of the patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/davidmalcolm'
closed_at = None
created_at = <Date 2011-07-21.21:02:13.493>
labels = []
title = 'Enhancements to gdb 7 debugging hooks'
updated_at = <Date 2012-08-03.23:38:38.895>
user = 'https://github.com/davidmalcolm'

bugs.python.org fields:

activity = <Date 2012-08-03.23:38:38.895>
actor = 'pitrou'
assignee = 'dmalcolm'
closed = False
closed_date = None
closer = None
components = ['Demos and Tools']
creation = <Date 2011-07-21.21:02:13.493>
creator = 'dmalcolm'
dependencies = []
files = ['22712', '22713']
hgrepos = []
issue_num = 12605
keywords = ['patch', 'needs review']
message_count = 10.0
messages = ['140828', '140829', '140993', '164109', '164113', '164124', '164168', '164182', '164183', '166423']
nosy_count = 9.0
nosy_names = ['loewis', 'barry', 'georg.brandl', 'gregory.p.smith', 'pitrou', 'vstinner', 'benjamin.peterson', 'dmalcolm', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue12605'
versions = ['Python 3.3']

[davidmalcolm]

I'm attaching patches to handle some more "events" in the gdb7 debugging hooks for CPython (aka Tools/gdb/libpython.py).

Currently, the hooks only care about C frames that are the bytecode interpreter (i.e. PyEval_EvalFrameEx)

This patch changes this, dividing C frames into:

• "python frames":
  • "bytecode frames" i.e. PyEval_EvalFrameEx
  • "other python frames": C frames that are of interest from a python
    perspective, but aren't bytecode
• everything else

so that the "py-bt", "py-up" and "py-down" commands will now work on the other kinds of "python" frames, in addition to the bytecode frames.

Specifically, the following new kinds of C frame are displayed:

• waiting on the GIL
• garbage-collection
• CFunctions (these weren't displayed before)

This should assist when debugging multithreaded crashes, to more easily get a sense of what every thread is doing.

Examples:
Showing a garbage-collection:
(gdb) py-bt
Garbage-collecting
<built-in method collect of module object at remote 0x7ffff18ea420>
File "<string>", line 4, in foo
File "<string>", line 6, in bar
File "<string>", line 7, in <module>

(gdb) py-bt-full
#1 Garbage-collecting
#2 <built-in method collect of module object at remote 0x7ffff18ea420>
#5 Frame 0x7ffff20d6d48, for file <string>, line 4, in foo ()
#8 Frame 0x7ffff20d6b70, for file <string>, line 6, in bar ()
#11 Frame 0x7ffff7fb1d80, for file <string>, line 7, in <module> ()

Showing an invocation of "time.sleep()":
(gdb) py-bt-full
#0 <built-in method sleep of module object at remote 0x7ffff1af8150>
#3 Frame 0x7ffff18aa060, for file <string>, line 3, in foo ()
#6 Frame 0x7ffff20d6b70, for file <string>, line 5, in bar ()
#9 Frame 0x7ffff7fb1d80, for file <string>, line 6, in <module> ()

Showing multiple threads, where all but one are waiting for the GIL:

Thread 5 (Thread 0x7fffeb5fe700 (LWP 10716)):
Traceback (most recent call first):
  Waiting for the GIL
  File "<string>", line 10, in run
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner
    self.run()
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap
    self._bootstrap_inner()

Thread 4 (Thread 0x7fffebfff700 (LWP 10715)):
Traceback (most recent call first):
  Waiting for the GIL
  File "<string>", line 10, in run
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner
    self.run()
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap
    self._bootstrap_inner()

Thread 3 (Thread 0x7ffff0dea700 (LWP 10714)):
Traceback (most recent call first):
  Waiting for the GIL
  File "<string>", line 10, in run
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner
    self.run()
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap
    self._bootstrap_inner()

Thread 2 (Thread 0x7ffff17eb700 (LWP 10713)):
Traceback (most recent call first):
  Waiting for the GIL
  File "<string>", line 10, in run
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 737, in _bootstrap_inner
    self.run()
  File "/home/david/coding/python-hg/cpython-more-gdb/build-debug/../Lib/threading.py", line 710, in _bootstrap
    self._bootstrap_inner()

Thread 1 (Thread 0x7ffff7fdb700 (LWP 10709)):
Traceback (most recent call first):
  File "<string>", line 18, in <module>

[davidmalcolm]

(On 2.7, I needed import_site=True to get the new tests to work from a fresh build: "import time" wasn't being found otherwise)

[pitrou]

The improvements are welcome, but I'm not sure they're ok for 3.2 or 2.7, since they're technically a new feature.
OTOH, one could argue that better debuggability trumps the no-feature rule.

[gpshead]

Better debuggability FTW! This is an update to Tools/gdb/ as such I'd like to see this make it into 3.3.

It doesn't touch the runtime or stdlib so I personally wouldn't consider this "adding a feature" and thus preventing its inclusion with 3.3 despite the beta process having started.

It'd the release manager's call. I'm bumping to release blocker for a decision from Georg. :)

[loewis]

I'm +1 on adding this in general, +0 on adding this to 3.3, and -0 on adding it to 2.7 right away.

I agree that the usual reasoning against new features doesn't apply here, since it's not a new Python (language or library) feature.

However, one concern is that it is a large change, and I predict that it will introduce some breakage regardless of what amount of code review goes into it. If the breakage can be detected during the betas, fine, but we need to consider that it may cause breakage of the entire python-gdb feature for some people after the release (which probably wouldn't be that terrible). Hence the +0 for 3.3.

For 2.7, it should only be added if it has survived in a prerelease branch for some time, hence the -0.

[birkenfeld]

Agreed with Martin, can't say I'm +1 on 3.3 either.

But I've just reviewed the patch, and it looks correct to me, so go ahead and I'll take the blame.

[pitrou]

Got the following error on Mageia 1:

======================================================================
FAIL: test_threads (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates threads that are waiting for the GIL
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/test/test_gdb.py", line 697, in test_threads
    cmds_after_breakpoint=['thread apply all py-bt'])
  File "/home/antoine/cpython/default/Lib/test/test_gdb.py", line 153, in get_stack_trace
    self.assertEqual(err, '')
AssertionError: "Error occurred in Python command: 'NoneType' object has no attribute 'startswit [truncated]... != ''
- Error occurred in Python command: 'NoneType' object has no attribute 'startswith'

[davidmalcolm]

I believe this was due to this line:
return self._gdbframe.name().startswith('pthread_cond_timedwait')
within is_waiting_for_gil(), and your gdb returned None for
self._gdbframe.name() for your build.

I've changed it to:
name = self._gdbframe.name()
if name:
return name.startswith('pthread_cond_timedwait')
in my latest version.

I've also bulletproofed against the --without-threads case by simply
skipping the cases that fail (some tests run into this gdb error:
Cannot find new threads: generic error
unless we add LD_PRELOAD=PATH-TO-libpthread.so.1 as a workaround, but it
seems that trying to add that workaround is more risky (what path?) than
simply skipping the tests under those circumstances.

Tested successfully on a Fedora 15 x86_64 box (gdb-7.3-43.fc15.x86_64)
with these configurations:

• --with-pydebug
• --with-pydebug --enable-shared
• (no flags = optimized)
• --enable-shared (optimized)
• --with-pydebug --without-threads

(Note that because of issue bpo-14774 I have to regenerate
_sysconfigdata.py between each configuration, otherwise it gets confused
about whether or not the build is optimized)

[python-dev]

New changeset abcd29c9a791 by David Malcolm in branch 'default':
Issue bpo-12605: Show information on more C frames within gdb backtraces
http://hg.python.org/cpython/rev/abcd29c9a791

[pitrou]

Dave, test_gdb fails consistently on the ARM buildbot:

======================================================================
FAIL: test_threads (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates threads that are waiting for the GIL
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/test/test_gdb.py", line 706, in test_threads
    self.assertIn('Waiting for the GIL', gdb_output)
AssertionError: 'Waiting for the GIL' not found in 'Breakpoint 1 at 0x9e36a: file Python/bltinmodule.c, line 962.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/arm-linux-gnueabi/libthread_db.so.1".\n[New Thread 0x2b2b2470 (LWP 8779)]\n[New Thread 0x2b5ff470 (LWP 8780)]\n[New Thread 0x2b8ff470 (LWP 8781)]\n[New Thread 0x2baff470 (LWP 8782)]\n\nBreakpoint 1, builtin_id (self=<module at remote 0x2aca5638>, v=42) at Python/bltinmodule.c:962\n962\t    return PyLong_FromVoidPtr(v);\n\nThread 5 (Thread 0x2baff470 (LWP 8782)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 639, in _bootstrap_inner\n    self.run()\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 616, in _bootstrap\n    self._bootstrap_inner()\n\nThread 4 (Thread 0x2b8ff470 (LWP 8781)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 639, in _bootstrap_inner\n    self.run()\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 616, in _bootstrap\n    self._bootstrap_inner()\n\nThread 3 (Thread 0x2b5ff470 (LWP 8780)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 639, in _bootstrap_inner\n    self.run()\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 616, in _bootstrap\n    self._bootstrap_inner()\n\nThread 2 (Thread 0x2b2b2470 (LWP 8779)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 639, in _bootstrap_inner\n    self.run()\n  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/threading.py", line 616, in _bootstrap\n    self._bootstrap_inner()\n\nThread 1 (Thread 0x2aac5300 (LWP 8776)):\nTraceback (most recent call first):\n  File "<string>", line 18, in <module>\n'

http://buildbot.python.org/all/builders/ARM%20Ubuntu%203.x