Repeated Py_Initialize and Py_Finalize usage brings fatal error?

[ocean-city]

BPO	9828
Nosy	@pitrou
Files	gildestroy.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2010-09-13.14:17:26.853>
created_at = <Date 2010-09-11.04:37:45.040>
labels = ['interpreter-core']
title = 'Repeated Py_Initialize and Py_Finalize usage brings fatal error?'
updated_at = <Date 2010-09-13.14:17:26.851>
user = 'https://bugs.python.org/ocean-city'

bugs.python.org fields:

activity = <Date 2010-09-13.14:17:26.851>
actor = 'pitrou'
assignee = 'pitrou'
closed = True
closed_date = <Date 2010-09-13.14:17:26.853>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2010-09-11.04:37:45.040>
creator = 'ocean-city'
dependencies = []
files = ['18861']
hgrepos = []
issue_num = 9828
keywords = ['patch']
message_count = 8.0
messages = ['116075', '116079', '116174', '116194', '116208', '116217', '116241', '116308']
nosy_count = 2.0
nosy_names = ['pitrou', 'ocean-city']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9828'
versions = ['Python 3.2']

[ocean-city]

When I ran the code in bpo-6869 (msg95444) on py3k, I noticed
Fatal Python error occurs.

# Actually, I need to copy created executable file into
# PC/VC6 directoly where python32_d.dll exists. Otherwise,
# the error "encoding.utf-8 was not found" occured even if
# PC/VC6 was added to environment variable PATH. But probably
# it is another issue.

E:\python-dev\py3k\PC\VC6>foo
--- 0 ---
[34859 refs]
--- 1 ---
Fatal Python error: drop_gil: wrong thread state

... And here is stacktrace.

_NMSG_WRITE(int 10) line 221
abort() line 44 + 7 bytes
Py_FatalError(const char * 0x1e371878) line 2109
drop_gil(_ts * 0x00920d90) line 321 + 10 bytes
PyEval_SaveThread() line 424 + 6 bytes
0012ff30()
PyCFunction_Call(_object * 0x00b99378, _object * 0x00afcc40, _object * 0x00000000) line 89 + 9 bytes
PyObject_Call(_object * 0x00b99378, _object * 0x00afcc40, _object * 0x00000000) line 2149 + 15 bytes
call_function_tail(_object * 0x00b99378, _object * 0x00afcc40) line 2181 + 15 bytes
_PyObject_CallMethod_SizeT(_object * 0x00bf6838, char * 0x1e332898, char * 0x00000000) line 2297 + 13 bytes
io_open(_object * 0x00c17f78, _object * 0x00ae0f88, _object * 0x00000000) line 423 + 16 bytes
PyCFunction_Call(_object * 0x00b0cab8, _object * 0x00ae0f88, _object * 0x00000000) line 84 + 15 bytes
PyObject_Call(_object * 0x00b0cab8, _object * 0x00ae0f88, _object * 0x00000000) line 2149 + 15 bytes
call_function_tail(_object * 0x00b0cab8, _object * 0x00ae0f88) line 2181 + 15 bytes
PyObject_CallMethod(_object * 0x00bf6138, char * 0x1e3d6abc, char * 0x1e3d6ab4) line 2258 + 13 bytes
create_stdio(_object * 0x00bf6138, int 0, int 0, char * 0x1e3d6968, char * 0x00000000, char * 0x00000000) line 808 + 48 bytes
initstdio() line 931 + 28 bytes
Py_InitializeEx(int 1) line 303 + 5 bytes
Py_Initialize() line 314 + 7 bytes
main() line 10 + 8 bytes
mainCRTStartup() line 206 + 25 bytes
KERNEL32! 77e789d5()

Because first run succeeded, Py_Finalize might not do enough
work to reset thread state, I guess.

I didn't try this on other platforms. (thread_nt.h was used here)

[ocean-city]

Sorry, msg92444

[pitrou]

I can't reproduce under Linux. Is 3.1 also affected?
Can you try the following patch:

Index: Python/pythonrun.c
===================================================================

--- Python/pythonrun.c	(révision 84718)
+++ Python/pythonrun.c	(copie de travail)
@@ -219,6 +219,8 @@
 
     /* auto-thread-state API, if available */
 #ifdef WITH_THREAD
+    if (PyEval_ThreadsInitialized())
+        PyEval_ReInitThreads();
     _PyGILState_Init(interp, tstate);
 #endif /* WITH_THREAD */

[pitrou]

It should be noted that repeatedly calling Py_Initialize / Py_Finalize leaks a lot (even without the "import _ctypes"):

--- 0 ---
[34543 refs]
--- 1 ---
[37540 refs]
--- 2 ---
[40504 refs]
--- 3 ---
[43468 refs]
--- 4 ---
[46432 refs]
--- 5 ---
[49396 refs]
--- 6 ---
[52360 refs]
--- 7 ---
[55324 refs]
--- 8 ---
[58288 refs]
--- 9 ---
[61252 refs]
--- 10 ---
[64216 refs]

And it also translates into memory consumption...

[ocean-city]

Is 3.1 also affected?
Sorry, 3.1 is not affected.

Can you try the following patch:
Your patch works. :-)

E:\python-dev\py3k\PC\VC6>ctypes_crash.exe
--- 0 ---
[36242 refs]
--- 1 ---
[39140 refs]
--- 2 ---
[41538 refs]
--- 3 ---
[43930 refs]
--- 4 ---
[46322 refs]
--- 5 ---
[48714 refs]
--- 6 ---
[51106 refs]
--- 7 ---
[53498 refs]
--- 8 ---
[55890 refs]
--- 9 ---
[58282 refs]

[pitrou]

Ok. How about this patch now?
(I'm trying to solve this in a clean way :))

[ocean-city]

gildestroy.patch also works.

E:\PYTHON~1\py3k\PC\VC6>ctypes_crash.exe
--- 0 ---
[36303 refs]
--- 1 ---
[39184 refs]
--- 2 ---
[41582 refs]
--- 3 ---
[43974 refs]
--- 4 ---
[46366 refs]
--- 5 ---
[48758 refs]
--- 6 ---
[51150 refs]
--- 7 ---
[53542 refs]
--- 8 ---
[55934 refs]
--- 9 ---
[58326 refs]

[pitrou]

Fix committed in r84774. Thank you!