Add a MemoryError singleton to fix an unlimited loop when the memory is exhausted

[vstinner]

BPO	19835
Nosy	@pitrou, @vstinner, @bitdancer, @serhiy-storchaka
Files	memerror_singleton.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-10-02.21:08:07.993>
created_at = <Date 2013-11-29.17:24:02.962>
labels = []
title = 'Add a MemoryError singleton to fix an unlimited loop when the memory is exhausted'
updated_at = <Date 2015-10-02.21:08:07.992>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-10-02.21:08:07.992>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-10-02.21:08:07.993>
closer = 'vstinner'
components = []
creation = <Date 2013-11-29.17:24:02.962>
creator = 'vstinner'
dependencies = []
files = ['32892']
hgrepos = []
issue_num = 19835
keywords = ['patch']
message_count = 5.0
messages = ['204742', '204744', '204745', '240340', '240353']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'Arfrever', 'r.david.murray', 'neologix', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue19835'
versions = ['Python 3.4']

[vstinner]

Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I propose to add a MemoryError read-only singleton to fix this corner case. Attributes cannot be modified, new attributes cannot be added. MemoryError attributes values:

• __cause__ = None
• __context__ = None
• __suppress_context__ = False
• __traceback__ = None
• args = ()

PyException_SetTraceback(), PyException_SetCause() and PyException_SetContext() do nothing when called on the MemoryError singleton.

A MemoryError can be raised on an integer overflow when computing the size of a memory block. In this case, you may still have available free memory and so you may be able to build a traceback object, chain exceptions, etc.

If you consider that attributes must be modifiable in this case, we can keep the MemoryError unchanged and keep the free list, but add a new special "ReadOnlyMemoryError" type which has exactly one instance, preallocated at startup (the singleton).

Read-only attributes are required to not keep references to objects when the MemoryError (singleton) is ne more used ("destroyed"): see for example test_memory_error_cleanup() of test_exceptions.

Python has already a PyExc_RecursionErrorInst singleton, which is used on recursion error. It is just a preallocated RuntimeError error, attributes are modifiable. Does this exception keep a traceback when it is no more used ("destroyed")?

Maybe using read-only attributes is overkill? Replacing the free list with a singleton is enough?

--

See also issue bpo-5437: before Python had a MemoryError singleton, but it was replaced by a free list to not keep references.

changeset: 65690:c6d86439aa91
branch: 3.1
parent: 65672:e4425d68dadf
user: Antoine Pitrou <solipsis@pitrou.net>
date: Thu Oct 28 23:06:57 2010 +0000
files: Include/pyerrors.h Lib/test/test_exceptions.py Misc/NEWS Objects/exceptions.c Python/errors.c
description:
Merged revisions 85896 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
r85896 | antoine.pitrou | 2010-10-29 00:56:58 +0200 (ven., 29 oct. 2010) | 4 lines

Issue bpo-5437: A preallocated MemoryError instance should not hold traceback
data (including local variables caught in the stack trace) alive infinitely.
........

--

Another option is maybe to clear frames of the traceback. Issue bpo-17934 added a clear() method to frame objects for that.

[vstinner]

Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I got this bug when I worked on the issue bpo-19817 which adds an arbitary limit to memory allocations using tracemalloc. I used this limit on the Python test suite to check how Python behaves under very low memory condition.

[pitrou]

Under very low memory condition, PyErr_NoMemory() or
PyErr_NormalizeException() enters an unlimited loop when the free list
of MemoryError becomes empty.

The real question is why the free list becomes empty.

Either way, I don't think a read-only singleton is a good idea. It may be simpler to call Py_FatalError in such cases.

[bitdancer]

Victor, do you still want to champion this, or shall we close it?

[vstinner]

Serhiy recently worked on MemoryError, maybe he wants to work on this issue?

I'm no more interested to work on this issue.