crash on badly initialised AttributeError

[scoder]

BPO	3443
Nosy	@amauryfa, @scoder

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-07-27.07:56:31.328>
created_at = <Date 2008-07-25.12:55:49.157>
labels = ['interpreter-core', 'invalid', 'type-crash']
title = 'crash on badly initialised AttributeError'
updated_at = <Date 2008-07-27.07:56:31.326>
user = 'https://github.com/scoder'

bugs.python.org fields:

activity = <Date 2008-07-27.07:56:31.326>
actor = 'amaury.forgeotdarc'
assignee = 'none'
closed = True
closed_date = <Date 2008-07-27.07:56:31.328>
closer = 'amaury.forgeotdarc'
components = ['Interpreter Core']
creation = <Date 2008-07-25.12:55:49.157>
creator = 'scoder'
dependencies = []
files = []
hgrepos = []
issue_num = 3443
keywords = []
message_count = 4.0
messages = ['70253', '70276', '70315', '70318']
nosy_count = 2.0
nosy_names = ['amaury.forgeotdarc', 'scoder']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue3443'
versions = ['Python 3.0']

[scoder]

I get a reproducible crash under Linux when running the test cases of
lxml's trunk in Py3b2. As usual with these things, it's not reproducible
when running the crashing test by itself, only when it hits the test
during the usual test run (which makes it look like somethings's leaking
between tests here).

Here is what gdb gives me so far.

----------------------
[...]
test_namespace_lookup (lxml.tests.test_classlookup.ClassLookupTestCase)
... ok
test_parser_based_lookup
(lxml.tests.test_classlookup.ClassLookupTestCase) ... ok
Doctest: test_css_select.txt ...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210656576 (LWP 938)]
0x080f3b89 in BaseException_str (self=0x8d747d4) at Objects/exceptions.c:88
88 switch (PyTuple_GET_SIZE(self->args)) {
(gdb) pyo self
object : AttributeError<NULL>
type : AttributeError
refcount: 10
address : 0x8d747d4
$1 = void
(gdb) print self->args
$2 = (PyObject *) 0x0
(gdb) bt
#0 0x080f3b89 in BaseException_str (self=0x8d747d4) at
Objects/exceptions.c:88
#1 0x0805b158 in PyObject_Str (v=0x8d747d4) at Objects/object.c:414
#2 0x0807951a in unicode_new (type=0x81523a0, args=0x8de77ac, kwds=0x0)
at Objects/unicodeobject.c:9247
#3 0x0806068d in type_call (type=0x81523a0, args=0x8de77ac, kwds=0x0)
at Objects/typeobject.c:636
#4 0x080d83a9 in PyObject_Call (func=0x81523a0, arg=0x8de77ac, kw=0x0)
at Objects/abstract.c:2178
#5 0x0808de50 in PyEval_EvalFrameEx (f=0x8e2d07c, throwflag=0) at
Python/ceval.c:3606
#6 0x0808fccd in PyEval_EvalFrameEx (f=0x8e2bf14, throwflag=0) at
Python/ceval.c:3481
#7 0x0808fccd in PyEval_EvalFrameEx (f=0x8e2cef4, throwflag=0) at
Python/ceval.c:3481
#8 0x0808fccd in PyEval_EvalFrameEx (f=0x8e2d4ec, throwflag=0) at
Python/ceval.c:3481
#9 0x08090f6b in PyEval_EvalCodeEx (co=0x8268458, globals=0xb7b772d4,
locals=0x0, args=0x8e2cea4, argcount=3, kws=0x8e2ceb0, kwcount=1,
defs=0x826c678, defcount=3, kwdefs=0x0,
closure=0x0) at Python/ceval.c:2830
#10 0x0808f63e in PyEval_EvalFrameEx (f=0x8e2cd54, throwflag=0) at
Python/ceval.c:3491
#11 0x0808fccd in PyEval_EvalFrameEx (f=0x8dc37a4, throwflag=0) at
Python/ceval.c:3481
#12 0x0808fccd in PyEval_EvalFrameEx (f=0x8da1d7c, throwflag=0) at
Python/ceval.c:3481
#13 0x08090f6b in PyEval_EvalCodeEx (co=0x84b21d0, globals=0x847d79c,
locals=0x0, args=0x8da0dbc, argcount=2, kws=0x8da0dc4, kwcount=2,
defs=0x8591b78, defcount=3, kwdefs=0x0,
closure=0x0) at Python/ceval.c:2830
#14 0x0808f63e in PyEval_EvalFrameEx (f=0x8da0c64, throwflag=0) at
Python/ceval.c:3491
#15 0x0808fccd in PyEval_EvalFrameEx (f=0x8da00ec, throwflag=0) at
Python/ceval.c:3481
#16 0x08090f6b in PyEval_EvalCodeEx (co=0xb7b80410, globals=0xb7b75824,
locals=0x0, args=0x8d6ee98, argcount=2, kws=0x89e8058, kwcount=0,
defs=0x826a518, defcount=1,
kwdefs=0x0, closure=0x0) at Python/ceval.c:2830
#17 0x080fe855 in function_call (func=0x8267e6c, arg=0x8d6ee8c,
kw=0x8ccee84) at Objects/funcobject.c:628
#18 0x080d83a9 in PyObject_Call (func=0x8267e6c, arg=0x8d6ee8c,
kw=0x8ccee84) at Objects/abstract.c:2178
#19 0x0808d73f in PyEval_EvalFrameEx (f=0x8d9ff84, throwflag=0) at
Python/ceval.c:3694
#20 0x08090f6b in PyEval_EvalCodeEx (co=0xb7b80458, globals=0xb7b75824,
locals=0x0, args=0x8d7f078, argcount=2, kws=0x0, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0,
closure=0x0) at Python/ceval.c:2830
[...]
(gdb) pyo 0x8268458
object : <code object print_exception at 0x8268458, file
"/usr/local/python3.0b2/lib/python3.0/traceback.py", line 136>
type : code
refcount: 2
address : 0x8268458
$3 = void
----------------------

When I call "pystack", gdb seems to lock up using 100% CPU, so I'm not
sure what else I can provide. The crash happening in a non-trivial
doctest makes is somewhat tricky to figure out what gets executed here.
At least, there is no anticipated AttributeError in the doctest, and it
doesn't seem to get raised when I run the test just by itself.

[amauryfa]

I reproduced the problem on Windows.
The exception shown is the AttributeError('next') raised and caught in
lxml._elementpath.find().
The "args" atribute is cleared in the BaseException_clear, during a call
to gc.collect() (in some tearDown() method).
Unfortunately this exception is still shomehow stored in the thread
state structure...

After some researches, I think that the problem is in Cython. Cython
tries to simulate a python frame without using the interpreter loop.
But at least an invariant is not respected: when a frame exits without
an exception set, the tstate->exc_type (and friends) should be NULL.
For example, at the end of the PyInit_etree() function (called by an
"import lxml.etree"), the tstate->exc_value contains an
AttributeError('module' object has no attribute 'unicode').

Now, the consequence may be that all these exceptions are "implicitely
chained" together. And there may be a subtle refcounting bug that shows
up only if this invariant is not respected.

[scoder]

Thanks a lot for the analysis. I was considering that this was a problem
with Cython, but since this was the first time I got a crash on this
(even Py3.0b1 didn't expose this), I wanted to ask here first.

Your explanation sounds like the right thing to do would be to clear the
exception state when a function exists cleanly but an exception was
raised and caught during its execution. So the exception state would
only stay available within the function itself. We could also try to
emulate the Py3 behaviour as outlined in PEP-3110. But we'll have to
discuss that on the Cython mailing list.

[amauryfa]

So the exception state would only stay available
within the function itself.
AFAIK That's what python does for caught exceptions.