-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Function-level import in os triggering an threaded import deadlock #44198
Comments
When I use subprocess.py from a child thread, sometimes it deadlocks. I #0 0x90024427 in semaphore_wait_signal_trap () and that the code in question is in os.py: def _execvpe(file, args, env=None):
from errno import ENOENT, ENOTDIR I think the problem is that since exec (the C function) hasn't yet been If I change os.py so that it imports the constants outside of This is on Mac OS X 10.4.8. |
Can you provide a test case or sample code that demonstrates this problem? I'm a bit unsure of if this really is a subprocess bug or a more general Python bug. |
I don't have time at the moment to write sample code that reproduces this. But, FYI, I was using PyObjC to create the threads. It might not happen with "threading" threads. And second, I think it's a bug in os.py, not in subprocess.py. Sorry for the confusion. |
Since both the reporter and I believes that this is not a bug in the subprocess module, I'm stepping back. |
I confirm that problem with deadlock on execution of PyImport_ImportModuleLevel exists. Here is a working example:
------------------------------------------------------------------ // get the global lock
PyEval_AcquireLock();
// get a reference to the PyInterpreterState
PyInterpreterState * mainInterpreterState = g_pyMainThreadState->interp;
ASSERT_KOBOLD( mainInterpreterState );
// create a thread state object for this thread
PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
// free the lock
PyEval_ReleaseLock();
return myThreadState;
}
---------------------------------------------------------------------
void DeadLock()
{
Py_Initialize();
PyEval_InitThreads();
g_AiGlobals = new Py::Dict();
g_pyInterpState = PyInterpreterState_New(); // save a pointer to the main PyThreadState object // release the lock
PyEval_ReleaseLock(); g_pyWorldThreadState = InitThreadScripting(); // import sys
// sys.path.append ("./scripts")
//
PyObject *p = PyImport_ImportModuleEx ("sys", **g_AiGlobals, NULL, NULL);
Py::Module mod_sys (p);
Py::List path = mod_sys.getAttr ("path");
path.append (Py::String ("scripts"));
path.append (Py::String ("scripts/sys"));
path.append (Py::String ("../../scripts"));
path.append (Py::String ("../../scripts/sys"));
Py_XDECREF (p);
// HERE IT OCCURS //
Log.ScriptsSrc("Python", "Running startup python scripts...");
PyObject *p = PyImport_ImportModuleEx ("startup", **g_AiGlobals, NULL, NULL); // <<< Here
if (reload) PyImport_ReloadModule (p);
Py::Module module (p);
Py_XDECREF (p);
} Execution locks right on PyImport_ImportModuleEx. Code from sturtup.py: import sys
from consts import * # Import of constants
import config as cfg # Import of configuration constants
reload(cfg) ################################################################################ # ---------------------------------------------------------------------------------------------------------------------------------------------------------------- class OurLogStdErr:
def write (self, txt):
printLog (txt, PRINT_ERROR)
# ----------------------------------------------------------------------------------------------------------------------------------------------------------------
class OurLogStdOut:
def write (self, txt):
printLog (txt) # ---------------------------------------------------------------------------------------------------------------------------------------------------------------- def CheckServerVersion():
# Checking for server build
if GetServerBuild() < MIN_SERVER_BUILD:
printLog( "YOU ARE TRYING TO RUN PYTHON SCRIPTS ON OUTDATED SERVER BUILD!\
\nREQUIRED SERVER BUILD: %s\
\nPlease Update your server core before running server!\
\nScripting Engine will be Shut Down!"\
% (MIN_SERVER_BUILD), PRINT_ERROR )
killScripting()
# ----------------------------------------------------------------------------------------------------------------------------------------------------------------
def GetScriptsVersion():
return SCRIPTS_VERSION ################################################################################ # Redirecting errors:
sys.stderr = OurLogStdErr()
# Redirecting output:
sys.stdout = OurLogStdOut() |
Do you have sample code that reproduces this problem? (Not necessarily code that has this problem 100% of the time) |
It seems the import lock is being triggered because of an import being Closing as "won't fix". |
I don't understand why the function-level imports cannot be removed. Wouldn't it be possible to do something like this: from errno import ENOENT as _ENOENT, ENOTDIR as _ENOTDIR
def _execvpe(file, args, env=None):
pass # Use _ENOENT and _ENOTDIR in this code BTW. it is IMO rather strange to close issues as wont fix when there is a |
First, because os is such a common module that hiding some uncommon Second, this is not a bug as the code is not behaving in an improper You can open the issue again if you want, but I think this is not worth |
Hi, |
Been thinking about it and as a compromise to people who view this as a |
Here's a preliminary fix (also see |
Checked in the patch to fix the forks-through-os.fork() cases, which |
I have a module that I was using on 2.5 that uses subprocess.Popen to monitor the output from some external programs in several different threads. Of course, subprocess.Popen uses os.fork. When I upgraded to 2.7 which includes this fix, this module ran into a deadlock since the fork is being executed from within an import. One could argue that my approach is poor style but one of the goals of this module is simplicity for the users....they simply need to import it to get the functionality of the module. Was this a desired side-effect? |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: