Interpreter aborts when chaining an infinite number of exceptions

[yury]

BPO	6028
Nosy	@rhettinger, @terryjreedy, @jcea, @amauryfa, @pitrou, @benjaminp, @asmeurer, @phmc
Files	stackoverflow.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-04-26.07:08:08.800>
created_at = <Date 2009-05-15.08:29:38.998>
labels = ['interpreter-core', 'type-bug']
title = 'Interpreter aborts when chaining an infinite number of exceptions'
updated_at = <Date 2017-03-25.04:17:26.306>
user = 'https://bugs.python.org/yury'

bugs.python.org fields:

activity = <Date 2017-03-25.04:17:26.306>
actor = 'Aaron.Meurer'
assignee = 'none'
closed = True
closed_date = <Date 2013-04-26.07:08:08.800>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2009-05-15.08:29:38.998>
creator = 'yury'
dependencies = []
files = ['13990']
hgrepos = []
issue_num = 6028
keywords = ['patch']
message_count = 14.0
messages = ['87797', '87804', '87824', '87836', '87856', '87867', '87890', '87891', '87898', '87907', '150241', '150243', '150250', '187835']
nosy_count = 10.0
nosy_names = ['rhettinger', 'terry.reedy', 'jcea', 'amaury.forgeotdarc', 'pitrou', 'benjamin.peterson', 'Arfrever', 'yury', 'Aaron.Meurer', 'pconnell']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6028'
versions = ['Python 3.2', 'Python 3.3']

[yury]

def error_handle():
    try:
        print(5/0)
    except:
        error_handle()

error_handle()

Fatal Python error: Cannot recover from stack overflow.
Aborted

The interpreter should not crash. Perhaps a RuntimeError should be
thrown instead.

[pitrou]

This is normal behaviour, actually. The RuntimeError *is* raised, but
you catch it in the except clause and then recurse again ad infinitum.
The interpreter realizes that it "cannot recover from stack overflow",
as the message says, and then bails out.

[amauryfa]

Hmm, the interpreter should not crash so easily with pyre Python code.
(The same code correctly raises RuntimeException wich python 2.x)
The issue should be corrected IMO.

The exact behavior seem to depend on where the recursion limit is detected:

• If it is detected at the beginning of the function (at the start of
  PyEval_EvalFrameEx) or before the "try" statement, the exception is
  correctly propagated and displayed. This case always happens with 2.x,
  and you get the same on 3.0 if you add for example str([[[[[[[[]]]]]]]])
  at the beginning of the function.

• The crash occurs when the recursion limit is detected while
  PyEval_EvalFrameEx calls PyErr_NormalizeException() (in the "why ==
  WHY_EXCEPTION" block). This does not change the control flow: the
  original exception is simply replaced by the RuntimeError, and nested
  calls continue 50 more frames.

Here is a tentative patch.

[pitrou]

Amaury, your patch might make some individual cases better, but it won't
prevent a FatalError from occurring in all cases.

Also, it makes things worse in the following case:

def recurse():
    try:
        recurse()
    except:
        recurse()
        
recurse()

(the script goes into an uninterruptible infinite loop, rather than
raising an explicit Fatal error)

More useful, IMHO, would be to patch Py_FatalError() so that a traceback
is printed.

[yury]

The code you posted causes an infinite loop in the 2.x branch as well.
Anyway, I do not see how crashing is a desired result. An infinite loop
means the programmer made a mistake somewhere. A crash means the
interpreter did.

[pitrou]

The code you posted causes an infinite loop in the 2.x branch as well.
Anyway, I do not see how crashing is a desired result.

I do not see what the "desired result" is in your example. The code is
obviously wrong. Did you get hit by that in production code or is it
just a proof-of-concept?

Also, this is not an actual crash (as in "segmentation fault"). It is
the interpreter /trying to protect itself from a crash/ which would be
caused by a stack overflow, and your code is trying to circumvent that
protection.

The fact that some errors cannot be recovered from is an unavoidable
fact of life. We may try to "fix" this particular case, but it will do
nothing for the more general case, so we might as well not "fix" it.

[yury]

I knew that python handles infinite recursion and gracefully errors out,
and I knew that exception chaining was new to 3.0, so I wanted to see if
they would work together. Apparently, they do not. Yet, the code works
fine in the 2.x branch. So, the 3.0 branch introduces a bug. I am not
sure how much clearer I can make this.

The code is in no way trying to circumvent anything. If there is about
to be a stack overflow, an exception show be raised. It should then
float up the stack. This is the desired result. In fact, this is the
only sane result. In an operating system, a program should not be able
to crash the entire system. In X, a client should not be able to crash
the server. Same principle applies here.

I am sorry I simply do not see your point. This seems so obvious. There
is a correct way of handling this error. You can safely recover from it.
If you insist on a general case, I have outlined it above.

I am not terribly thrilled about your attitude of "we might as well
leave it broken." Why bother working on the project at all?

[pitrou]

While we seem to disagree on whether this a real bug (and I'll leave it
at that), I'll just stress once again that a "fatal error" is totally
different from an uncontrolled crash like a segmentation fault -- as I
explained and although you don't seem to understand, the "fatal error"
mechanism is there /precisely/ to avoid uncontrolled crashes. Also, your
analogy with processes crashing an X server is flawed. You should be in
control of all the code running in your own program; Python doesn't
offer any mechanism to protect good code from badly written code when
running in the same address space -- any such expectation is misguided.

(for various ways of producing fatal errors, please search for
"Py_FatalError()" in the source tree)

[amauryfa]

You should be in control of all the code running in your own program
This is not the case with applications that embed Python to provide
users a way to script the application.
All the usages of Py_FatalError I've seen detect programming errors at
the C level, or at interpreter startup.
I still take the rule that no python code should be able to crash the
program unexpectedly.

[benjaminp]

Well, perhaps something like bpo-1195571 should be added.

[terryjreedy]

I believe bpo-3555, bpo-7338, and *13644 are basically duplicates of this issue. I have left this one open because it has a try at a patch. I think any patch should be tested with the other examples.

I agree with Antoine that an intentional exit is not a crash.
I also agree that the current procedure is not really a bug either. According to the language spec, the interpreter should recurse forever;-), just like "while True: pass" iterates 'forever'. Given that it does not due to finite limitations, the exact alternate behavior is undefined.

Now, if someone can find a way to better handle infinite recursion mixed with exceptions, without re-introducing real crashes or eliminating the benefits of the 3.0 changes, great.

Yury, I think Antoine's point is that gracefully handling all the different kinds of programming mistakes in a finite system is a delicate and difficult balancing act.

[yury]

Rather than aborting with a stack overflow, I feel it is more natural to raise an exception. If it is not too difficult to implement, perhaps another type of exception should be raised. Since chained exceptions are new to 3.x, there should be a new exception to describe errors that happen in chaining. Perhaps stopping chaining at a certain depth and truncating all further exceptions with a blanket "ChainingException" exception. Or perhaps truncating the chain and wrapping it in another exception before returning it to user code.

While this is my proposed solution, I apologize I cannot volunteer to write the patch. The time investment on my part would involve learning most of the working of the interpreter. Perhaps someone who is already familiar with it would be interested.

[rhettinger]

FWIW, I concur with Antoine on this one.

[pitrou]

Closing as won't fix. There is no sane way around the current behaviour.