crash with unbounded recursion in except statement

[xxm]

BPO	42500
Nosy	@gpshead, @ronaldoussoren, @ericvsmith, @ambv, @markshannon, @serhiy-storchaka, @vedgar
PRs	bpo-42500: Fix recursion in or after except #23568 [3.9] bpo-42500: Fix recursion in or after except (GH-23568) #24501 [3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 #25179
Superseder	bpo-45806: Cannot Recover From StackOverflow in 3.9 Tests
Files	error1.py: run this program by using command " python error1.py" under Ubuntu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/markshannon'
closed_at = <Date 2020-12-02.13:54:51.328>
created_at = <Date 2020-11-29.07:39:23.532>
labels = ['interpreter-core', '3.9', 'type-crash']
title = 'crash with unbounded recursion in except statement'
updated_at = <Date 2021-11-19.18:16:43.116>
user = 'https://bugs.python.org/xxm'

bugs.python.org fields:

activity = <Date 2021-11-19.18:16:43.116>
actor = 'lukasz.langa'
assignee = 'Mark.Shannon'
closed = True
closed_date = <Date 2020-12-02.13:54:51.328>
closer = 'Mark.Shannon'
components = ['Interpreter Core']
creation = <Date 2020-11-29.07:39:23.532>
creator = 'xxm'
dependencies = []
files = ['49635']
hgrepos = []
issue_num = 42500
keywords = ['patch']
message_count = 12.0
messages = ['382042', '382055', '382057', '382059', '382069', '382070', '382113', '382115', '382138', '382311', '387905', '390182']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'ronaldoussoren', 'eric.smith', 'lukasz.langa', 'Mark.Shannon', 'serhiy.storchaka', 'veky', 'xxm']
pr_nums = ['23568', '24501', '25179']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '45806'
type = 'crash'
url = 'https://bugs.python.org/issue42500'
versions = ['Python 3.9']

[xxm]

This program can work well on Python 3.5.2 and Python2.7 with the following output.
"Invalid Entry, try again"

However it will crash on Python3.9.0rc1, 3.10.0a2 with the following error message:
"
Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007fc4b679b700 (most recent call first):
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 38 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
File "/home/xxm/Desktop/methodfuzzer/error/error1.py", line 39 in status
...
Aborted (core dumped)
"

[ronaldoussoren]

The crash is not nice, but....

The script contains the following line in the definition of status():

except status() as e:

That will cause unbounded recursion when os.fstat raises an exception, which likely happens here. You probably want to catch OSError (or os.error) instead.

This scriptlet is a smaller reproduction of the problem:

def status():

try:
1/0

except status():
pass

status()

[vedgar]

Recursion limit is probably set too high, but nonetheless, the program is nonsensical, the line 39 especially.

The reason for difference in behavior is that RecursionError is caught by blank except in the earlier Pythons, while it crashes the stack in later ones. In any case, it's almost certain this isn't what you wanted to write.

(And yet another reason to remove blank except from Python: it abuse to correct use ratio is probably over two orders of magnitude.)

[ronaldoussoren]

The value of the recursion limit is not relevant here, I get the same crash when I set the recursion limit to 100 (in my reproducer scriptlet).

There might be a missing recursion level check in the exception handling machinery though, self-recursion in an except-clause is rather unusual. But that's just a blind guess, I haven't though much about what is supposed to happen here.

[ericvsmith]

Here's the smallest reproducer I could come up with. It fails on Windows with 3.9 native(compiled locally) (Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow), works (raises RecursionError) with cygwin 3.8.3.

import os

def status():
    for fd in range(4):
        try:
            st = os.fstat(fd)
        except status() as e:
            pass

status()

[ericvsmith]

Note that changing the os.fstat line to just "raise OSError()" no longer causes the "Fatal Python error", but rather gives the expected recursion exception.

Here's a shorter version that causes the fatal error in Windows native 3.9, cygwin 3.8.3, and Fedora Linux 3.7.7. So this isn't new with 3.9.

import os

def status():
    try:
        st = os.fstat(4)
    except status() as e:
        pass

status()

[ronaldoussoren]

As mentioned earlier I can reproduce this without calling os.fstat (python 3.9.1rc, macOS 10.15):

# ---

def status():
   try:
      1/0

except status():
pass

status()
# 

---

The crash happens both with the default recursion limit as well with a recursion limit of 10. Setting the recursion limit doesn't seem to have an effect on what's happening (in both cases I get a long traceback).