New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
threading.local() does not work with C-created threads #50876
Comments
When threads are created by a C extension loaded with ctypes, More information and example program showing the behaviour: |
Here is a simple testcase: $ gcc -fPIC -g -c -Wall lib.c
$ gcc -shared -Wl -o lib.so lib.o
$ python test.py
In callback: creating thread
Python callback_fn called, setting lc.x = 42
Python callback_fn called, setting lc.x = 42 Expected output: $ python test.py
In callback: creating thread
Python callback_fn called, setting lc.x = 42
Python callback_fn called, lc.x = 42 |
Note also that if the from __future__ import is enabled, the program segfaults instead (separate bug)? |
As far as I know, the thread creation done in the file is not correct. While creating threads in C extension, there are certain rules to follow. Firstly, Python should be made thread-aware if it is not already i.e. call PyEval_InitThreads in the C callback function. After its creation, the thread should bootstrap to be able to execute Python code. It should create a new PyThreadState for itself by calling PyThreadState_New. For this, the thread should be passed the InterpreterState* through the entry function. Before executing any Python code, the thread should make sure that the current ThreadState * is corresponding to it by calling PyEval_RestoreThread. |
Swapnil's analysis looks correct to me - there are certain rules you have to follow before calling back into the Python interpreter core. If you don't follow them, the behaviour you will get is completely undefined. If the problem still occurs even when the C thread is correctly initialised for calling into the Python C API then this issue can be reopened. |
One more thing: This explains the difference between threading.local and _threading_local:
|
@Swapnil: the rules you quote are correct for the C extension, but do not apply when using ctypes, because ctypes is doing the required initializations automatically. However, if Amaury is correct, ctypes performs the initializations in a way that break the threading.local functionality. I think the best way to address this bug would therefore be to add a warning to the ctypes documentation that C created threads will not support threading.local(). |
Not quite. ctypes (or more exactly: the PyGILState_Ensure() and PyGILState_Release() functions, which are the standard API to do this) is in a situation where it must call Python code from a thread which has no PyThreadState. So it creates a thread state, runs the code, and destroys the thread state; is this wrong? If you want to keep Python state during your C thread, there are 4 lines to add to your function: void *async_cb(void *dummy)
{
PyGILState_STATE gstate = PyGILState_Ensure();
Py_BEGIN_ALLOW_THREADS
(*callback_fn)();
(*callback_fn)();
Py_END_ALLOW_THREADS
PyGILState_Release(gstate);
pthread_exit(NULL);
} |
No, I am not saying that the behaviour of ctypes is wrong. It just happens to have some effects on threading.local that I think should be documented. That's why I reassigned this as a documentation bug. Please reconsider closing this bug. I'm also happy to change the type to "Feature request". As an aside: I think in most cases one uses ctypes to call functions that are already compiled, so changing the source is not an option. |
To be a bit more constructive, why not add something like this in paragraph to http://docs.python.org/library/ctypes.html#callback-functions: "Note that if the callback function is called in a new thread that has been created outside of Python's control (i.e., by the foreign code that calls the callback), ctypes creates a new dummy Python thread on every invocation. That means that values stored with |
This is not specific to ctypes. |
One point of ctypes is to save the user the trouble of having to create a full blown C extension, so I don't think it's reasonable to expect a ctypes user to have read the full C API documentation as well. Only a very small subset of the page that you gave is actually relevant for use with ctypes. Why not put this information where a ctypes user can find it easily? |
The suggestion in http://bugs.python.org/issue6627#msg116722 is a good one. This a case where the user may legitimately not realise that threading.local is stored in the *temporary* state created by ctypes rather than in something more persistent inside the interpreter. Since the ctypes state is per callback, it won't persist across calls, even when they're made from the same thread. Suggested wording: |
Nick, the last statement, |
On Tue, Sep 21, 2010 at 2:39 PM, Swapnil Talekar <report@bugs.python.org> wrote:
The shorter version doesn't mean the same thing though - the ctypes
There's no easy way to make the thread state persist between calls, as |
Here's a patch that (I think) incorporates all the comments. If someone could apply it, that would be great :-). |
(adding the documentation and ctypes experts from http://docs.python.org/devguide/experts.html to noisy list in the hope to get this moving again.) |
New changeset f4eade5df217 by Benjamin Peterson in branch '3.3': New changeset 9cd2d7a3f9f2 by Benjamin Peterson in branch '2.7': New changeset fd647825475a by Benjamin Peterson in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: