Abort in PyErr_PrintEx() when no memory

[xdegaye]

BPO	30817
Nosy	@vstinner, @xdegaye, @serhiy-storchaka
PRs	bpo-30817: Fix PyErr_PrintEx() when no memory #2526 [3.6] bpo-30817: Fix PyErr_PrintEx() when no memory (GH-2526). #4107
Files	memerr.py pyerr_printex.patch set-last-exception.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-24.14:44:13.263>
created_at = <Date 2017-06-30.14:19:31.165>
labels = ['interpreter-core', '3.7', 'type-crash']
title = 'Abort in PyErr_PrintEx() when no memory'
updated_at = <Date 2017-10-24.14:44:13.262>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2017-10-24.14:44:13.262>
actor = 'xdegaye'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-24.14:44:13.263>
closer = 'xdegaye'
components = ['Interpreter Core']
creation = <Date 2017-06-30.14:19:31.165>
creator = 'xdegaye'
dependencies = []
files = ['46982', '46983', '46986']
hgrepos = []
issue_num = 30817
keywords = ['patch']
message_count = 9.0
messages = ['297408', '297411', '297414', '297483', '297495', '297508', '304809', '304818', '304913']
nosy_count = 3.0
nosy_names = ['vstinner', 'xdegaye', 'serhiy.storchaka']
pr_nums = ['2526', '4107']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue30817'
versions = ['Python 3.6', 'Python 3.7']

[xdegaye]

To reproduce the abort (the set_nomemory() function is being added to _testcapi in bpo-30695):

$ ./python memerr.py
python: Objects/call.c:89: _PyObject_FastCallDict: Assertion `!PyErr_Occurred()' failed.
Aborted (core dumped)

The corresponding backtrace:

#0 0x00007ffff7131670 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff7132d00 in abort () from /usr/lib/libc.so.6
#2 0x00007ffff712a45a in __assert_fail_base () from /usr/lib/libc.so.6
#3 0x00007ffff712a4d2 in __assert_fail () from /usr/lib/libc.so.6
#4 0x00000000004628ce in _PyObject_FastCallDict (
callable=<built-in method excepthook of module object at remote 0x7ffff70ab8d8>,
args=args@entry=0x7fffffffe330, nargs=nargs@entry=3, kwargs=kwargs@entry=0x0)
at Objects/call.c:89
#5 0x0000000000426004 in PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1)
at Python/pythonrun.c:641
#6 0x00000000004263ac in PyErr_Print () at Python/pythonrun.c:510
#7 0x0000000000427586 in PyRun_SimpleFileExFlags (fp=fp@entry=0x987620, filename=<optimized out>,
filename@entry=0x7ffff6febcd0 "memerr.py", closeit=closeit@entry=1,
flags=flags@entry=0x7fffffffe4cc) at Python/pythonrun.c:403
#8 0x0000000000427673 in PyRun_AnyFileExFlags (fp=fp@entry=0x987620,
filename=0x7ffff6febcd0 "memerr.py", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe4cc)
at Python/pythonrun.c:82
#9 0x0000000000437217 in run_file (fp=fp@entry=0x987620, filename=0x91a190 L"memerr.py",
p_cf=p_cf@entry=0x7fffffffe4cc) at Modules/main.c:341
#10 0x0000000000437bae in Py_Main (argc=argc@entry=2, argv=argv@entry=0x918020)
at Modules/main.c:901
#11 0x000000000041d7fa in main (argc=2, argv=0x7fffffffe6c8) at ./Programs/python.c:102

[xdegaye]

With pyerr_printex.patch we get the following correct results (rc=120 is set by Py_Main() after Py_FinalizeEx() returns with an error):

$ ./python memerr.py 5 12
set_nomemory(0, 5)
result = _PythonRunResult(rc=1, out=b'', err=b'MemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')

=============================

set_nomemory(0, 6)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nMemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')
=============================
set_nomemory(0, 7)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nMemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')
=============================
set_nomemory(0, 8)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : MemoryError()\ntype    : MemoryError\nrefcount: 1\naddress : 0x7f7ae1be3eb0\nlost sys.stderr')
=============================
set_nomemory(0, 9)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f8aead15eb0\nlost sys.stderr')
=============================
set_nomemory(0, 10)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f7b489d1eb0\nlost sys.stderr')
=============================
set_nomemory(0, 11)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7ff97365eeb0\nlost sys.stderr')
=============================
set_nomemory(0, 12)
result = _PythonRunResult(rc=120, out=b'', err=b"sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f559c9e0eb0\nlost sys.stderr\nException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>\nMemoryError")

=============================

[vstinner]

pyerr_printex.patch LGTM, please convert it to a GitHub PR.

[serhiy-storchaka]

Is it good to ignore errors of setting sys attributes? Can we prevent these errors?

[xdegaye]

There are just 6 cases left where the return code of _PySys_SetObjectId() is ignored:

Python/pylifecycle.c|689 col 5| _PySys_SetObjectId(&PyId_stderr, pstderr);
Python/pylifecycle.c|1211 col 9| _PySys_SetObjectId(&PyId_stderr, pstderr);
Python/pylifecycle.c|1657 col 5| _PySys_SetObjectId(&PyId_stdin, std);
Python/pylifecycle.c|1666 col 5| _PySys_SetObjectId(&PyId_stdout, std);
./Python/pythonrun.c|104 col 9| _PySys_SetObjectId(&PyId_ps1, v = PyUnicode_FromString(">>> "));
./Python/pythonrun.c|109 col 9| _PySys_SetObjectId(&PyId_ps2, v = PyUnicode_FromString("... "));

Error should be handled there in the same way errors are handled within the functions where they are invoked (returning -1, NULL or aborting). This should be the purpose of another issue as it is acceptable I think that there is no test case for those changes.

I am leaving the day after tomorrow for few weeks, without internet access (sailing to the mid-atlantic), and will create then a new issue if no one else does that first.

[serhiy-storchaka]

I tried to set last_type, last_value and last_traceback to None at startup, and this also fixes a crash. But this changes behavior (these attributes became set), and deleting them doesn't guaranties that following setting will be successful, especially with new compact dict implementation.

[xdegaye]

Removing 2.7 as the problem cannot be reproduced here. PyEval_CallObjectWithKeywords() does not assert on PyErr_Occurred().

[xdegaye]

New changeset 66caacf by xdegaye in branch 'master':
bpo-30817: Fix PyErr_PrintEx() when no memory (bpo-2526)
66caacf

[xdegaye]

New changeset d5d7954 by xdegaye in branch '3.6':
[3.6] bpo-30817: Fix PyErr_PrintEx() when no memory (GH-2526). (bpo-4107)
d5d7954