Segfault provoked by generators and exceptions #44139
Comments
|
A reproducible segfault when using heavily-nested Unfortunately, I haven't yet been able to provoke this The gist of the code is a series of nested generators |
klaas
mannequin
added
the
interpreter-core
klaas
mannequin
added
the
interpreter-core
|
Logged In: YES Program received signal SIGSEGV, Segmentation fault. #39 0x080bff32 in PyEval_EvalCode (co=0xb7f397b8,
globals=0xb7f6ca44, locals=0xb7f6ca44) at Python/ceval.c:494
#40 0x080ddff1 in PyRun_FileExFlags (fp=0x98a4008,
filename=0xbfffd4a3 "scoreserver.py", start=257,
globals=0xb7f6ca44, locals=0xb7f6ca44, closeit=1,
flags=0xbfffd298) at Python/pythonrun.c:1264
#41 0x080de321 in PyRun_SimpleFileExFlags (fp=Variable "fp"
is not available.
) at Python/pythonrun.c:870
#42 0x08056ac4 in Py_Main (argc=1, argv=0xbfffd334) at
Modules/main.c:496
#43 0x00a69d5f in __libc_start_main () from /lib/libc.so.6
#44 0x08056051 in _start () |
|
Logged In: YES I've produced a simplified traceback with a single generator The relevant call in traceback.c is: and I can verify that oldtb contains garbage: #0 0x080e4296 in PyTraceBack_Here (frame=0x8964d94) at #19 0x080bfda3 in PyEval_EvalCodeEx (co=0xb7f6ade8,
globals=0xb7fafa44, locals=0x0,
args=0xb7cc5ff8, argcount=1, kws=0x0, kwcount=0,
defs=0x0, defcount=0, closure=0x0)
at Python/ceval.c:2833
#20 0x08104083 in function_call (func=0xb7cc7294,
arg=0xb7cc5fec, kw=0x0)
at Objects/funcobject.c:517
#21 0x0805a660 in PyObject_Call (func=0xb7cc7294,
arg=0xb7cc5fec, kw=0x0)
at Objects/abstract.c:1860 |
|
Logged In: YES I cannot yet produce an only-python script which reproduces def getdocs():
def f():
<some somehwat time-consuming operation>
while True:
f()
yield None----------------------------------------------------------------------------- class B(object):
def __init__(self,):
pass
def doit(self):
# must be an instance var to trigger segfault
self.docIter = getdocs()
print self.docIter # this is the generator
referred-to in the traceback
for i, item in enumerate(self.docIter):
if i > 9:
break
print 'exiting generator'
class A(object):
""" Process entry point / main thread """
def __init__(self):
while True:
try:
self.func()
except Exception, e:
print 'right after raise'
def func(self):
b = B()
thread = threading.Thread(target=b.doit)
thread.start()
start_t = time.time()
while True:
try:
if time.time() - start_t > 1:
raise Exception
except Exception:
print 'right before raise'
# SIGSEGV here. If this is changed to
# 'break', no segfault occurs
raise
if __name__ == '__main__':
A() |
|
Logged In: YES Can you please review/try attached patch? Can anybody tell |
|
Logged In: YES
As documented (in concrete.tex), PyGen_New(f) steals a """
In short, that PyGen_New() doesn't incref the frame passed It's possible that the intent is flawed ;-), but offhand I |
|
Logged In: YES Despite Tim's reassurrance, I'm afraid that Martin's patch |
|
Logged In: YES I've attached a much simplified pure-Python script (hope.py) exiting generator at most twice before crapping out. At the time, the Note that this is not a debug-build obmalloc gimmick! This So somebody is accessing a thread state here after it's been Note that a PyThreadState (a frame's f_tstate) is /not/ a |
|
Logged In: YES
This bit is fairly easy: they are allocated without the GIL being held, which No idea about the real problem, sadly. |
|
Logged In: YES Mike, what platform are you having the problem on? I tried Tim's hope.py on Linux x86_64 and Mac OS X 10.4 with Invalid read of size 8 at 0x4CEBFE: PyTraceBack_Here (traceback.c:117) by 0x49C1F1: PyEval_EvalFrameEx (ceval.c:2515) by 0x4F615D: gen_send_ex (genobject.c:82) by 0x4F6326: gen_close (genobject.c:128) by 0x4F645E: gen_del (genobject.c:163) by 0x4F5F00: gen_dealloc (genobject.c:31) by 0x44D207: _Py_Dealloc (object.c:1928) by 0x44534E: dict_dealloc (dictobject.c:801) by 0x44D207: _Py_Dealloc (object.c:1928) by 0x4664FF: subtype_dealloc (typeobject.c:686) by 0x44D207: _Py_Dealloc (object.c:1928) by 0x42325D: instancemethod_dealloc (classobject.c:2287) Address 0x56550C0 is 88 bytes inside a block of size 152 by 0x4C3899: tstate_delete_common (pystate.c:256) by 0x4C3926: PyThreadState_DeleteCurrent (pystate.c:282) by 0x4D4043: t_bootstrap (threadmodule.c:448) by 0x4B24C48: pthread_start_thread (in The only way I can think to fix this is to keep a set of |
|
Logged In: YES
Neal, note that it's the /Windows/ malloc that fills freed The valgrind report is pointing at the same thing. Whether |
|
We are experiencing the same segfault in our application, reliably. Running our unit test suite just segfault everytime on both Linux and Mac OS X. Applying Martin's patch fixes the segfault, and makes everything fine and dandy, at the cost of some memory leaks if I understand properly. This particular bug prevents us to upgrade to python 2.5 in production. |
|
The following patch resets the thread state of the generator when it is resumed, which prevents the segfault for me: Index: Objects/genobject.c --- Objects/genobject.c (revision 52849)
+++ Objects/genobject.c (working copy)
@@ -77,6 +77,7 @@
Py_XINCREF(tstate->frame);
assert(f->f_back == NULL);
f->f_back = tstate->frame;
+ f->f_tstate = tstate;
gen->gi_running = 1;
result = PyEval_EvalFrameEx(f, exc); |
|
This fixes the segfault problem that I was able to reliably reproduce on Linux. We need to get this applied (assuming it is the correct fix) to the source to make Python 2.5 usable for me in production code. |
|
Why do frame objects have a thread state in the first place? In particular, why does PyTraceBack_Here get the thread state from the frame, instead of using the current thread? Introduction of f_tstate goes back to r7882, but it is not clear why it was done that way. |
|
Bumping priority to see if this should go into 2.5.1. |
|
I don't like mklaas' patch, since I think it is conceptually wrong to have PyTraceBack_Here() use the frame's thread state (mklaas describes it as dirty, and I agree). I'm proposing an alternative patch (tr.diff); please test this as well. |
|
A quick test on code that always segfaulted with unpatched Python 2.5 seems to work. |
|
This is now fixed in r53531 and r53532. For the trunk, it is likely that f_tstate will get eliminated altogether in the near future. People who had the problem are really encouraged to test 2.5.1c1 when it is released. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: