PyOS_CheckStack does not work

[amauryfa]

BPO	3996
Nosy	@loewis, @amauryfa, @kristjanvalur, @vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/amauryfa'
closed_at = <Date 2008-11-22.20:07:20.148>
created_at = <Date 2008-09-29.11:34:10.161>
labels = ['OS-windows']
title = 'PyOS_CheckStack does not work'
updated_at = <Date 2008-11-22.20:07:20.143>
user = 'https://github.com/amauryfa'

bugs.python.org fields:

activity = <Date 2008-11-22.20:07:20.143>
actor = 'amaury.forgeotdarc'
assignee = 'amaury.forgeotdarc'
closed = True
closed_date = <Date 2008-11-22.20:07:20.148>
closer = 'amaury.forgeotdarc'
components = ['Windows']
creation = <Date 2008-09-29.11:34:10.161>
creator = 'amaury.forgeotdarc'
dependencies = []
files = []
hgrepos = []
issue_num = 3996
keywords = ['patch']
message_count = 6.0
messages = ['74024', '74044', '74052', '75365', '76254', '76257']
nosy_count = 4.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'kristjan.jonsson', 'vstinner']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue3996'
versions = ['Python 2.6']

[amauryfa]

On Windows, PyOS_CheckStack is supposed to protect the interpreter from
stack overflow. But doing this, it always crashes when the stack is
nearly full.

The reason is a bad check of the return value of _resetstkoflw():
according to MSDN, the return value is "Nonzero if the function
succeeds, zero if it fails.":
http://msdn.microsoft.com/en-us/library/89f73td2.aspx

The patch below is enough to replace the "Fatal Python error: Could not
reset the stack!" into a "MemoryError: stack overflow" exception.

Tested with:
>>> loop = None,
>>> for x in xrange(100000): loop = {'x': loop}
...
>>> len(repr(loop))

Index: Python/pythonrun.c
===================================================================

--- Python/pythonrun.c  (revision 66486)
+++ Python/pythonrun.c  (working copy)
@@ -1749,7 +1755,7 @@
                        EXCEPTION_EXECUTE_HANDLER :
                        EXCEPTION_CONTINUE_SEARCH) {
                int errcode = _resetstkoflw();
-               if (errcode)
+               if (errcode == 0)
                {
                        Py_FatalError("Could not reset the stack!");
                }

[vstinner]

This issue may be related: bpo-1069092

[amauryfa]

Yes, bpo-1069092 is another case where PyOS_CheckStack is exercised.
But this other issue was first reported on Unix, when PyOS_CheckStack is
currently implemented only for Windows.

[vstinner]

See also issue bpo-3999: I wrote a new generic handler for stack
overflow "exception".

[loewis]

The patch is fine, please apply it to all versions from 2.6 to 3.0.

[amauryfa]

Committed r67343 (trunk) and r67344 (release26-maint)