python-gdb error: Python Exception <class 'RuntimeError'> Type does not have a target

[DylanCali]

BPO	35132
Nosy	@vstinner, @lisroach
PRs	bpo-35132: Fixes missing target in gdb pep0393 check. #11848 [3.7] bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848) #12284

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-12.09:22:05.318>
created_at = <Date 2018-11-01.10:40:19.368>
labels = ['3.8', 'type-bug', '3.7']
title = "python-gdb error: Python Exception <class 'RuntimeError'> Type does not have a target"
updated_at = <Date 2019-03-13.00:16:18.937>
user = 'https://bugs.python.org/DylanCali'

bugs.python.org fields:

activity = <Date 2019-03-13.00:16:18.937>
actor = 'Dylan Cali'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-12.09:22:05.318>
closer = 'vstinner'
components = ['Demos and Tools']
creation = <Date 2018-11-01.10:40:19.368>
creator = 'Dylan Cali'
dependencies = []
files = []
hgrepos = []
issue_num = 35132
keywords = ['patch']
message_count = 9.0
messages = ['329052', '334196', '335518', '335519', '335618', '337713', '337729', '337730', '337824']
nosy_count = 3.0
nosy_names = ['vstinner', 'lisroach', 'Dylan Cali']
pr_nums = ['11848', '12284']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35132'
versions = ['Python 3.7', 'Python 3.8']

[DylanCali]

Python version: 3.6.6-debug
System: Kernel: 4.15.0-38-generic x86_64
Distro: Linux Mint 18.2 Sonya
CPU: Quad core Intel Xeon E3-1505M
Memory: 32018.6MB

Expected:

py-list and py-bt to print the current python frame and traceback when attaching to a hung python process with gdb, a debug build of python, and cpython/Tools/gdb/libpython.py loaded.

Actual:

py-list and py-bt fail with:

Python Exception <class 'RuntimeError'> Type does not have a target.:
Error occurred in Python command: Type does not have a target.

Invoking 'set python print-stack full' in gdb produces the more useful:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "~/git/cpython/Tools/gdb/libpython.py", line 916, in filename
      File "~/git/cpython/Tools/gdb/libpython.py", line 1158, in proxyval
    RuntimeError: Type does not have a target.

so it is failing at:

    fields = gdb.lookup_type('PyUnicodeObject').target().fields()

in libpython.py [1].

Reproduce:

I haven't been able to create a simple standalone program that triggers the failure. I am working on a PR for Keras to try and fix the multiprocessing support in some of their utility classes. Currently the tests are sporadically hanging and my intention was to use python's gdb integration to identify exactly where and why the hangs are occuring... but I can't get that information at the moment because of the error above when invoking py-list and py-bt.

So, unfortunately, the shortest path to reproduce is to checkout the PR branch, run the offending tests, connect with gdb, and invoke py-list/py-bt:

* install a debug version of 3.6.6 if one isn't already available
* git clone https://github.com/calid/keras.git -b fix-multiprocessing-hang
* cd keras
* pip install -e .[tests]
* pip install tensorflow
* py.test tests/keras/utils/data_utils_test.py
* wait for hang
* gdb -p <parent_pid_of_harness>
* invoke py-list or py-bt

I am happy to poke around in libpython.py and try to fix/submit a PR myself, but I'm not at all familiar with the python internals so I would need some guidance. And obviously let me know if this isn't actually a bug but rather environment related/user error.

Thank you!

[1] https://github.com/python/cpython/blob/v3.6.6/Tools/gdb/libpython.py#L1158

[lisroach]

I experienced this bug as well and have tried to dig into it a little.

I experimented with a service I have that uses shared libraries. If I compile with high level of optimizations for the C++ code (-O3) I don't experience the issue, but compiling without any optimizations I do.

I also tried with a script that does not use any shared libraries and I do not see the issue.

Digging into it a little, when running with the optimized version I found:

gdb.lookup_type('PyUnicodeObject')

returns a gdb.TYPE_CODE_TYPEDEF, which is the correct type. Running with the non-optimized version instead returns a gdb.TYPE_CODE_STRUCT.

The struct type errors because it has no target(), but I actually think target() might be superfluous here.

Both versions if I only call fields = gdb.lookup_type('PyUnicodeObject').fields() returns a list of fields, one of which is named data for the purposes of the pep check, so target() doesn't seem vital.

[vstinner]

Distro: Linux Mint 18.2 Sonya

Hum, Dylan: what is your gdb version?

On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless:

$ gdb ./python
GNU gdb (GDB) Fedora 8.2-6.fc29
...
(gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').target().fields()])
['_base', 'data']

(gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').fields()])
['_base', 'data']

I tested on a Python compiled manually from source, I ran my test in the directory of Python source code.

I also tested on /usr/bin/python3 (system Python), same behavior.

[vstinner]

Python 3.6 no longer accept bugfixes, only security fixes:
http://devguide.python.org/#status-of-python-branches

You can try faulthandler has a workaround. For example, install faulthandler signal handler using faulthandler.register(signal.SIGUSR1) and then send a SIGUSR1 signal to the blocked process. You might want to write the output into a file if you don't have access the program stdout: use the 'file' parameter of faulthandler.register(). See also faulthandler.dump_traceback_later() to use a timeout.

https://docs.python.org/dev/library/faulthandler.html

[DylanCali]

Hum, Dylan: what is your gdb version?

$ gdb --version
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless

I can confirm removing the .target() call in libpython.py resolved the issue for me with no ill effects

You can try faulthandler has a workaround.

I'll give this a try, thank you

[lisroach]

New changeset 1ceb3a3 by Lisa Roach in branch 'master':
bpo-35132: Fixes missing target in gdb PEP-0393 check. (GH-11848)
1ceb3a3

[vstinner]

New changeset 047f8f2 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-35132: Fixes missing target in gdb PEP-0393 check. (GH-11848) (GH-12284)
047f8f2

[vstinner]

Thanks Dylan Cali for the bug report. Lisa Roach fixed the bug in 3.7 and master branches. Until a new release is published, you can copy manually the file from:
https://github.com/python/cpython/blob/3.7/Tools/gdb/libpython.py

[DylanCali]

Thank you! And thank you Lisa Roach for the investigation and fix.