invalid exception context

[vstinner]

BPO	3611
Nosy	@amauryfa, @pitrou, @vstinner, @benjaminp
Files	lostcontext.py 3611.patch lostcontext2.py pyerr_setobject_reentrant.patch: Fix PyErr_SetObject() to be re-entrant pyerr_setobject_reentrant-v2.patch: New version of my previous patch: fix PyErr_SetObject() to be re-entrant pyerr_seterror_protect.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-08-29.07:14:24.613>
created_at = <Date 2008-08-20.02:33:18.833>
labels = ['interpreter-core', 'type-crash', 'release-blocker']
title = 'invalid exception context'
updated_at = <Date 2008-08-29.07:14:24.595>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2008-08-29.07:14:24.595>
actor = 'amaury.forgeotdarc'
assignee = 'none'
closed = True
closed_date = <Date 2008-08-29.07:14:24.613>
closer = 'amaury.forgeotdarc'
components = ['Interpreter Core']
creation = <Date 2008-08-20.02:33:18.833>
creator = 'vstinner'
dependencies = []
files = ['11180', '11182', '11196', '11201', '11203', '11207']
hgrepos = []
issue_num = 3611
keywords = ['patch', 'needs review']
message_count = 41.0
messages = ['71503', '71505', '71506', '71522', '71548', '71551', '71552', '71553', '71557', '71558', '71569', '71570', '71574', '71575', '71579', '71587', '71589', '71591', '71592', '71593', '71594', '71595', '71596', '71657', '71658', '71659', '71661', '71663', '71664', '71665', '71698', '71704', '71711', '71739', '71740', '71741', '71793', '71794', '72041', '72043', '72120']
nosy_count = 4.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'vstinner', 'benjamin.peterson']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue3611'
versions = ['Python 3.0']

[vstinner]

After few seconds (30 sec to 5 minutes), my program (Fusil) crashs at:

PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at Python/ceval.c:2459
    Py_CLEAR(tstate->exc_traceback);

It crashs because tstate->exc_traceback points to a zombi object:
{_ob_next = 0xdbdbdbdb, _ob_prev = 0xdbdbdbdb, ob_refcnt = -606348326,
ob_type = 0xdbdbdbdb}

(refcnt is 0xdbdbdbdb-1)

I'm using py3k rev 65882 compiled with CFLAGS "-O0 -ggdb"
and --with-pydebug. Sorry, I don't have more informations yet and I
can't explain how to reproduce the bug :-/

[vstinner]

The crash is on ProcError exception raising (line 21):

20      except IOError as err:
21          raise ProcError("Unable to open %r: %s" % (filename, 

---

(gdb) where
#0 0x0805bcc7 in PyObject_Hash (v=0x8691034) at Objects/object.c:796
#1 0x08062125 in set_contains_key (so=0x856ec6c, key=0x8691034) at
Objects/setobject.c:682
#2 0x0806659f in PySet_Contains (anyset=0x856ec6c, key=0x8691034) at
Objects/setobject.c:2263
#3 0x080def3a in print_exception_recursive (f=0xb7bea654,
value=0x8656814, seen=0x856ec6c) at Python/pythonrun.c:1432
#4 0x080df0ea in PyErr_Display (exception=0x84542ac, value=0x8656814,
tb=0x865bc34) at Python/pythonrun.c:1470
#5 0x080e68f6 in sys_excepthook (self=0xb7e013b4, args=0x865b334) at
Python/sysmodule.c:119
#6 0x08161d29 in PyCFunction_Call (func=0xb7e016c4, arg=0x865b334,
kw=0x0) at Objects/methodobject.c:81
#7 0x08118cc5 in PyObject_Call (func=0xb7e016c4, arg=0x865b334,
kw=0x0) at Objects/abstract.c:2181
#8 0x080b2ea8 in PyEval_CallObjectWithKeywords (func=0xb7e016c4,
arg=0x865b334, kw=0x0) at Python/ceval.c:3283
#9 0x080de463 in PyErr_PrintEx (set_sys_last_vars=1) at
Python/pythonrun.c:1258
#10 0x080de06f in PyErr_Print () at Python/pythonrun.c:1150
#11 0x080e0007 in Py_FatalError (msg=0xbf9a1498 "Python/ceval.c:2459
object at 0x865b6b4 has negative ref count -606348326")
at Python/pythonrun.c:1863
#12 0x0805a821 in _Py_NegativeRefcount
(fname=0x8197098 "Python/ceval.c", lineno=2459, op=0x865b6b4) at
Objects/object.c:194
#13 0x080b004f in PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at
Python/ceval.c:2459
(...)

(gdb) frame 13
#13 0x080b004f in PyEval_EvalFrameEx (f=0x85c9794, throwflag=0) at 
Python/ceval.c:2459
2459                                            
Py_CLEAR(tstate->exc_traceback);

(gdb) pystack
/home/haypo/ptrace/ptrace/linux_proc.py (21): openProc
/home/haypo/ptrace/ptrace/linux_proc.py (24): readProc
/home/haypo/ptrace/ptrace/linux_proc.py (31): readProcessProc
/home/haypo/ptrace/ptrace/linux_proc.py (74): readProcessStat
(...)

---

linux_proc.py code near line 21:
(...)
7 class ProcError(Exception):
8 pass
(...)
16 def openProc(path):
17 try:
18 filename = procFilename(path)
19 return open(filename)
20 except IOError as err:
21 raise ProcError("Unable to open %r: %s" % (filename,
err))
(...)
29 def readProcessProc(pid, key):
30 try:
31 return readProc(path_join(str(pid), str(key)))
32 except ProcError as error:
33 raise ProcError("Process %s doesn't exist: %s" % (
34 pid, error))
(...)
73 def readProcessStat(pid):
74 stat = readProcessProc(pid, 'stat')
75 return ProcessState(stat)
76

[vstinner]

I also noticed a crash in PyErr_SetObject(): in block <line 68..86>,
tstate->exc_value value may changes and so it's possible that
tstate->exc_value becomes NULL. I added a test to avoid this crash:

Index: Python/errors.c
===================================================================

--- Python/errors.c     (révision 65899)
+++ Python/errors.c     (copie de travail)
@@ -88,7 +88,7 @@
                   This is O(chain length) but context chains are
                   usually very short. Sensitive readers may try
                   to inline the call to PyException_GetContext. */
-               if (tstate->exc_value != value) {
+               if (tstate->exc_value != value && tstate->exc_value != 
NULL) {
                        PyObject *o = tstate->exc_value, *context;
                        while ((context = PyException_GetContext(o))) 
{
                                Py_DECREF(context);

[pitrou]

I also noticed a crash in PyErr_SetObject(): in block <line 68..86>,
tstate->exc_value value may changes and so it's possible that
tstate->exc_value becomes NULL.

How can tstate->exc_value become NULL at that point? Could you investigate?

[vstinner]

How can tstate->exc_value become NULL at that point? [error.c:86]
Could you investigate?

PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since
the same thread state is the same, tstate->exc_value also changes
during this call.

[pitrou]

Selon STINNER Victor <report@bugs.python.org>:

> How can tstate->exc_value become NULL at that point? [error.c:86]
> Could you investigate?

PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since
the same thread state is the same, tstate->exc_value also changes
during this call.

Hmm, indeed.
However, I don't see why your Python code triggers that. Does instantiating the
ProcError object somehow fail, and for what reason? Or did you witness it in
other circumstances?

As for the original problem ("tstate->exc_traceback points to a zombi object")
it would be nice to have a small snippet of Python code to reproduce it. Is it
possible for you to do that?

[pitrou]

I retract what I said above. PyErr_SetObject() and friends only change
tstate->curexc_* variables, not tstate->exc_*.

The former (tstate->curexc_) contain the current, pending (uncaught)
exception state of the thread. The latter (tstate->exc_) contain the
exception currently handled by an "except" statement in Python code.

So, theoretically, there is no way calling PyEval_CallObject() can
change tstate->exc_value:

• if it raises an exception or calls code that raises an exception
  without catching it, the exception ends up in tstate->curexc_, not in
  tstate->exc_
• if it calls code that raises an exception /and catches it/, exception
  stacking means the exception should be discarded after the end of the
  "except" block and the last tstate->exc_* values restored.

Of course, there may be bugs in the implementation of the above. The
best way to find out would be to have a small snippet of Python code
reproducing the problem. :-)

[vstinner]

About the PyEval_CallObject() call in errors.c, here is an example:
Call exception<0x81dcee0>(args<0x8751dc4>)
with exception=
object : <class 'AttributeError'>
type : type
refcount: 6
address : 0x81dcee0
and args=
object : ("type object 'ProcError' has no
attribute '__subclasscheck__'",)
type : tuple
refcount: 1
address : 0x8751dc4

This exception may comes from PyObject_IsSubclass() which calls
PyObject_GetAttr(cls, "__subclasscheck__").

[pitrou]

[...]

This exception may comes from PyObject_IsSubclass() which calls
PyObject_GetAttr(cls, "__subclasscheck__").

If you look at the code for PyObject_IsSubclass(), there is a pair of
PyErr_Fetch / PyErr_Restore calls around PyObject_GetAttr(cls,
"__subclasscheck__"). So this exception should vanish immediately rather than
stay around.

[vstinner]

I'm unable to write a short Python script to reproduce the bug. So
download the full program:
http://neudorf.hachoir.org/tmp/fusil3000.tar.gz

Run in with:
$ gdb python3.0
(gdb) run fusil-python --fast

[amauryfa]

The problem comes when PyErr_SetObject triggers garbage collection which
runs python code (finalizers...).

I could reproduce the crash several times, and each time garbage
collection was triggered by the normalization of the exception (the call
to PyEval_CallObject(exception, args)).

An obvious fix is to save exc_value near the "/* Implicit exception
chaining */" comment.

However, I fear that there are other places where exc_value is reset,
and "exception chaining" may break in subtle ways.

[pitrou]

The problem comes when PyErr_SetObject triggers garbage collection which
runs python code (finalizers...).

Mmmh, normally this shouldn't change the value of tstate->exc_value once
that Python code returns. That's what exception stacking is for.

Having a snippet deterministically reproducing the problem would really
help in any case.

An obvious fix is to save exc_value near the "/* Implicit exception
chaining */" comment.

Well, it may be a fix for the crash but I'm not sure it makes the
semantics correct. If tstate->exc_value was really changed, there is a
reason and I'm not sure it should be ignored.

[amauryfa]

Having a snippet deterministically reproducing the problem
would really help in any case.
Here you are. Don't ask how I found this.

The attached script, when run, prints

(KeyError(), ValueError())
(KeyError(), None)

The current exception context (tstate->exc_value) is lost when a
generator is deleted, if it was paused inside a "try" block.

This is the cause of the crash: the gc runs inside PyErr_SetObject and
collects such a generator.