test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output #82420

vstinner · 2019-09-20T21:32:40Z

BPO	38239
Nosy	@vstinner, @zooba, @stratakis, @miss-islington
PRs	bpo-38239: Fix test_gdb for Link Time Optimization (LTO) #16422 [3.8] bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) #16424 [3.7] bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) #16426

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-27.12:53:10.213>
created_at = <Date 2019-09-20.21:32:40.269>
labels = ['3.8', '3.7', 'tests', '3.9']
title = 'test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output'
updated_at = <Date 2019-09-27.12:53:10.210>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-09-27.12:53:10.210>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-27.12:53:10.213>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-09-20.21:32:40.269>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38239
keywords = ['patch']
message_count = 16.0
messages = ['352896', '352897', '352910', '352949', '353012', '353020', '353023', '353121', '353122', '353127', '353313', '353317', '353318', '353321', '353325', '353357']
nosy_count = 4.0
nosy_names = ['vstinner', 'steve.dower', 'cstratak', 'miss-islington']
pr_nums = ['16422', '16424', '16426']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38239'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

vstinner · 2019-09-20T21:32:39Z

AMD64 RHEL8 LTO 3.x:
https://buildbot.python.org/all/#/builders/312/builds/4

FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)
FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)
FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)
FAIL: test_bytes (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)
FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)
FAIL: test_int (test.test_gdb.PrettyPrintTests)
FAIL: test_lists (test.test_gdb.PrettyPrintTests)
FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_sets (test.test_gdb.PrettyPrintTests)
FAIL: test_singletons (test.test_gdb.PrettyPrintTests)
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
FAIL: test_truncation (test.test_gdb.PrettyPrintTests)
FAIL: test_tuples (test.test_gdb.PrettyPrintTests)

Example of failure:

======================================================================
FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 515, in test_NULL_ob_type
    self.assertSane('id(42)',
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 486, in assertSane
    self.get_gdb_repr(source,
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 261, in get_gdb_repr
    self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113\n1113\tPyLong_FromVoidPtr(void *p)\n#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113\n#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442\n#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96\n#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984\n#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496\n#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296\n#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712\n#7  PyEval_EvalCode (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714\n#8  run_eval_code_obj (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117\n#9  run_mod (mod=<optimized out>, filename=\'<string>\', globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139\n#10 PyRun_StringFlags (flags=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\\n") at Python/pythonrun.c:1008\n#11 PyRun_SimpleStringFlags (command="id(42)\\n", flags=) at Python/pythonrun.c:460\n#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260\n#13 pymain_run_python (exitcode=) at Modules/main.c:558\n#14 Py_RunMain () at Modules/main.c:646\n#15 pymain_main (args=) at Modules/main.c:676\n#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700\n#17 __libc_start_main () from /lib64/libc.so.6\n#18 _start () at Python/ceval.c:5444\n'
Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113
1113 PyLong_FromVoidPtr(void *p)
#0 builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113
#1 cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442
#2 _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96
#3 call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984
#4 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496
#5 _PyEval_EvalCodeWithName (co=<code at remote 0x7ffff7f13920>, globals={'__name_': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296
#6 PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712
#7 PyEval_EvalCode (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714
#8 run_eval_code_obj (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117
#9 run_mod (mod=<optimized out>, filename='<string>', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139
#10 PyRun_StringFlags (flags=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\n") at Python/pythonrun.c:1008
#11 PyRun_SimpleStringFlags (command="id(42)\n", flags=) at Python/pythonrun.c:460
#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260
#13 pymain_run_python (exitcode=) at Modules/main.c:558
#14 Py_RunMain () at Modules/main.c:646
#15 pymain_main (args=) at Modules/main.c:676
#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700
#17 __libc_start_main () from /lib64/libc.so.6
#18 _start () at Python/ceval.c:5444

vstinner · 2019-09-20T21:35:31Z

Similar errors on:

AMD64 RHEL8 LTO + PGO 3.7:
https://buildbot.python.org/all/#/builders/255/builds/13

AMD64 RHEL8 LTO 3.7:
https://buildbot.python.org/all/#/builders/293/builds/13

AMD64 RHEL8 LTO 3.8:
https://buildbot.python.org/all/#/builders/265/builds/12

vstinner · 2019-09-20T23:41:48Z

Similar issue on AMD64 Fedora Stable LTO 3.8:
https://buildbot.python.org/all/#/builders/235/builds/2

zooba · 2019-09-21T18:31:47Z

Shouldn't this test be suppressed on LTO builds?

#0 builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113

builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

re.match(r'.*#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)\s+at\s+\S*Python/bltinmodule.c.*' ...

vstinner · 2019-09-23T12:25:35Z

AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
https://buildbot.python.org/all/#/builders/252/builds/2

And AMD64 Fedora Stable LTO 3.7:
https://buildbot.python.org/all/#/builders/270/builds/2

vstinner · 2019-09-23T14:34:21Z

builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

gdb may be confused by LTO. I'm not sure if how LTO works to debug symbols.

Shouldn't this test be suppressed on LTO builds?

Maybe. I don't know.

zooba · 2019-09-23T14:48:10Z

Greg seems to have done LTO related things in the past, maybe he knows? (All my PGO contributions only apply to Windows :) )

gpshead · 2019-09-24T22:11:53Z

The test probably has a fragile regex. I don't spend time with gdb things much myself.

zooba · 2019-09-24T22:38:20Z

Um, okay then.

The test already includes this:

if ((sysconfig.get_config_var('PGO_PROF_USE_FLAG') or 'xxx') in
(sysconfig.get_config_var('PY_CORE_CFLAGS') or '')):
raise unittest.SkipTest("test_gdb is not reliable on PGO builds")

Apparently this no longer works. Who understands these variables enough to fix the issue?

vstinner · 2019-09-25T00:41:18Z

Apparently this no longer works. Who understands these variables enough to fix the issue?

Some failures are LTO + PGO builds, but test_gdb also fails on LTO without PGO builds.

The PGO_PROF_USE_FLAG check only checks for PGO.

vstinner · 2019-09-26T14:54:28Z

New changeset 64b4a3a by Victor Stinner in branch 'master':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
64b4a3a

vstinner · 2019-09-26T15:08:28Z

Python 2.7 doesn't seem to be affected: I failed to reproduce the test_gdb failure on Python 2.7 with LTO.

miss-islington · 2019-09-26T15:13:43Z

New changeset c989340 by Miss Islington (bot) in branch '3.8':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
c989340

vstinner · 2019-09-26T15:30:19Z

New changeset e6b5ed1 by Victor Stinner in branch '3.7':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) (GH-16426)
e6b5ed1

vstinner · 2019-09-26T16:17:23Z

AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

vstinner · 2019-09-27T12:53:10Z

AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

I checked an all buildbots mentioned in this issues are back to green (success). I close the issue.

AMD64 RHEL8 LTO 3.x:
https://buildbot.python.org/all/#/builders/312/builds/4

Green

AMD64 RHEL8 LTO + PGO 3.7:
https://buildbot.python.org/all/#/builders/255/builds/13

Green

AMD64 RHEL8 LTO 3.7:
https://buildbot.python.org/all/#/builders/293/builds/13

Green

AMD64 RHEL8 LTO 3.8:
https://buildbot.python.org/all/#/builders/265/builds/12

Green

Similar issue on AMD64 Fedora Stable LTO 3.8:
https://buildbot.python.org/all/#/builders/235/builds/2

Green

AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
https://buildbot.python.org/all/#/builders/252/builds/2

Green

And AMD64 Fedora Stable LTO 3.7:
https://buildbot.python.org/all/#/builders/270/builds/2

Green

vstinner added 3.9 only security fixes tests Tests in the Lib/test dir labels Sep 20, 2019

vstinner changed the title ~~test_gdb fails on AMD64 RHEL8 LTO 3.x: Unexpected gdb output~~ test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output Sep 20, 2019

vstinner added 3.7 (EOL) end of life 3.8 (EOL) end of life labels Sep 26, 2019

vstinner closed this as completed Sep 27, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output #82420

test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output #82420

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

zooba commented Sep 21, 2019

vstinner commented Sep 23, 2019

vstinner commented Sep 23, 2019

zooba commented Sep 23, 2019

gpshead commented Sep 24, 2019

zooba commented Sep 24, 2019

vstinner commented Sep 25, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

miss-islington commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 27, 2019

test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output #82420

test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output #82420

Comments

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

vstinner commented Sep 20, 2019

zooba commented Sep 21, 2019

vstinner commented Sep 23, 2019

vstinner commented Sep 23, 2019

zooba commented Sep 23, 2019

gpshead commented Sep 24, 2019

zooba commented Sep 24, 2019

vstinner commented Sep 25, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

miss-islington commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 27, 2019