Fatal Python error "XXX block stack overflow" when exception stacks >10

[myzhang1029]

BPO	39934
Nosy	@ambv, @markshannon, @serhiy-storchaka, @pablogsal, @myzhang1029, @iritkatriel
PRs	bpo-39934: Account for control blocks in 'except' in compiler. #22395 [3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) #23303
Files	exception_nest.py: Run exception_nest.py and run the generated python code

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-05.10:53:27.428>
created_at = <Date 2020-03-11.13:24:52.447>
labels = ['interpreter-core', '3.10', '3.8', '3.9', 'type-crash']
title = 'Fatal Python error "XXX block stack overflow" when exception stacks >10'
updated_at = <Date 2021-05-05.10:53:27.427>
user = 'https://github.com/myzhang1029'

bugs.python.org fields:

activity = <Date 2021-05-05.10:53:27.427>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-05.10:53:27.428>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2020-03-11.13:24:52.447>
creator = 'myzhang1029'
dependencies = []
files = ['48969']
hgrepos = []
issue_num = 39934
keywords = ['patch']
message_count = 18.0
messages = ['363914', '377433', '377434', '377437', '377439', '377440', '377441', '377444', '377445', '377490', '378178', '378192', '381280', '385035', '385037', '385039', '385042', '393002']
nosy_count = 6.0
nosy_names = ['lukasz.langa', 'Mark.Shannon', 'serhiy.storchaka', 'pablogsal', 'myzhang1029', 'iritkatriel']
pr_nums = ['22395', '23303']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39934'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[myzhang1029]

I apologize for describing this issue badly, but I'll try anyway.
The code to demonstrate the issue is attached, so it might be better to read that instead.
I noticed that when more than 10 exceptions are raised sequentially (i.e. one from another or one during the handling of another), the interpreter crashes saying "Fatal Python error: XXX block stack overflow".
This happens in python 3.7, 3.8 and development(git 39c3493) versions, but not in python2.7. Using ipython also fixes this issue.
I know this case is rare, but the maximum number of recursions is more than 2000, and the maximum number of statically nested blocks sepcified in frameobject.c is 20, so I'm pretty sure this isn't intended behavior.

[iritkatriel]

The error is coming from here:

cpython/Objects/frameobject.c

Line 958 in cb9879b

Py_FatalError("block stack overflow");

and CO_MAXBLOCKS is defined in Include/cpython/code.h
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */

This is not about recursion or about exception, it's about static nesting level.

There is an example here showing an input that gives the same error with >20 nested while blocks:

cpython/Lib/test/test_syntax.py

Line 545 in c8f29ad

[iritkatriel]

In summary, I think this is not-a-bug.

[serhiy-storchaka]

It is a bug. Compiler explicitly checks if the number of nested "try" blocks does not exceed the limit of CO_MAXBLOCKS, but it does not count implicit "try" blocks inserted when your assign an exception in the "except" clause.

try:
    ...
except Exception as e:
    ...

is actually translated to

try:
    ...
except Exception:
    try:
        e = ...
        ...
    finally:
        e = None
        del e

So we have double number of nested "try" blocks.

[serhiy-storchaka]

Humm, my supposition was not absolutely correct. The cause is that the compiler and the interpreter use the stack of size CO_MAXBLOCKS for different things. The interpreter pushes a thing for the "except" clause, while the compiler does not do it.

[iritkatriel]

Unlike the 22 nested while,

if 1:
  if 2:
    ...
    if 22:
        pass

doesn't error. Is that correct?

[iritkatriel]

Another oddity:

This gives the error:
for x in '1':
for x in '2':
for x in '3':
for x in '4':
for x in '5':
for x in '6':
for x in '8':
for x in '9':
for x in '10':
for x in '11':
for x in '12':
for x in '13':
for x in '14':
for x in '15':
for x in '16':
for x in '17':
for x in '18':
for x in '19':
for x in '20':
for x in '21':
for x in '22':
pass

but this doesn't:

for x in '1':
 for x in '2':
  for x in '3':
   for x in '4':
    for x in '5':
     for x in '6':
      for x in '8':
       for x in '9':
        for x in '10':
         for x in '11':
          for x in '12':
           for x in '13':
            for x in '14':
             for x in '15':
              for x in '16':
               for x in '17':
                for x in '18':
                 for x in '19':
                  for x in '20':
                   for x in '21':
                    pass
                   else:
                    for x in '22':
                     pass

[markshannon]

iritkatriel

What error do you see and on what version?

[iritkatriel]

On windows 10, master:

python.bat --version
Running Release|Win32 interpreter...
Python 3.10.0a0

python.bat x.py
Running Release|Win32 interpreter...
SyntaxError: too many statically nested blocks

That's when x.py has the nested for loops without else. The error goes away if I add the else.

[markshannon]

New changeset 02d126a by Mark Shannon in branch 'master':
bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395)
02d126a

[iritkatriel]

After studying Mark's patch and the rest of the code, I understand the for loop oddity.

The else block of the for loop comes after the
compiler_pop_fblock(c, FOR_LOOP, start);

So you can do this all day and it won't complain:

for x in '1': pass
else:
 for x in '2': pass
 else:
  for x in '3': pass
  else:
   for x in '4': pass
   else:
    for x in '5': pass
    else:
     for x in '6': pass
     else:
      for x in '7': pass
      else:
       for x in '8': pass
       else:
        for x in '9': pass
        else:
         for x in '10': pass
         else:
          for x in '11': pass
          else:
           for x in '12': pass
           else:
            for x in '13': pass
            else:
             for x in '14': pass
             else:
              for x in '15': pass
              else:
               for x in '16': pass
               else:
                for x in '17': pass
                else:
                 for x in '18': pass
                 else:
                  for x in '19': pass
                  else:
                   for x in '20': pass
                   else:
                    for x in '21': pass
                    else:
                     for x in '22': pass
                     else:
                      for x in '23': pass
                      else:
                       for x in '24': pass
                       else:
                        for x in '25': pass
                        else:
                         for x in '26': pass
                         else:
                          for x in '27': pass
                          else:
                           for x in '28': pass
                           else:
                            for x in '29': pass

I guess the same goes for while loops, and the else of a try-except.

Since If blocks were deliberately left out of this game, I'm assuming that this "static nesting" is actually number of frames, rather than true static nesting. If that is the case then there is no issue here and we can close this ticket.

But if this is something to be fixed, then I am happy to make a patch along the lines of Mark's (which I partially already have).

[pablogsal]

PR 22395 should be backported to 3.8 and 3.9

[markshannon]

New changeset 48a9c0e by Irit Katriel in branch '3.9':
[3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) (GH-23303)
48a9c0e

[markshannon]

Does this need backporting to 3.8, or is 3.9 sufficient?

[iritkatriel]

There were additional merge conflicts when I tried to create a manual 3.8 backport, more significant than the 3.9 ones IIRC.

[markshannon]

Pablo, are you OK closing this without a 3.8 backport?

[pablogsal]

Pablo, are you OK closing this without a 3.8 backport?

I would be supportive, but we should check with Łukasz as he is the release manager of 3.8

[iritkatriel]

This was fixed in 3.9+ and 3.8 is in security fix now, so closing.