PyEval_SaveThread: NULL tstate #46
Comments
|
This type of error usually means something isn't compiled correctly or there is a version mismatch between mod_python and the Python libs being used, hard to check without some debugging. |
|
I am using it with python 2.7.10. then configure mod_python as let me know if i can provide you with some more debug information |
|
I'm running into the same problem. (gentoo system) -- rebuilt both apache and mod_python Any suggestions for debugging or providing additional information? |
|
grisha you said it require debugging but you haven't provided what you need for debugging. I an pretty sure me and Cheyenne can provide you logs. |
|
@natsuFairytail Sorry - with my schedule I haven't been able to devote any time to this, but the kind of debugging I meant was just probably attaching to the httpd process with gdb and figuring out why tstate is null. This error isn't generated by mod_python, it's coming from the CPython code, so you'll need to have all of httpd, Python and mod_python compiled with debugging enabled, which might be a little involved if you haven't done this before, sorry! |
|
If there is something specific, I can do the debugging (including setting debugging for both python and apache). If you have some specific "break points" or memory values you would like to see let me know. |
|
Just doing a quick debug (with debug symbols for apache, mod_python and CPython) here is the bt at the SIGABRT. If you can let me know what to look for I can go digging |
|
|
|
Both Python and mod_python are compiled with WITH_THREAD I was looking at the mod_python code, especially the comments around the use of the initialized variable. Could it be that the python framework got unloaded? So the idea with the PyEval_SaveThread is that it releases the Python GIL. Could this be a case where the GIL wasn't held because of the reload. -OR- could it be that PyThreadState global_tstate is getting wiped during the module reload? -- is there something that I can "poke" at in gdb to determine this for you? |
|
There is no chance of anything happening between line 780 ( PyEval_InitThreads: https://github.com/python/cpython/blob/2.7/Python/ceval.c#L249 Another possibility is that somehow mod_python is initialized more than once, and |
|
So stepping through the code in python_init we get to the test in line 771 the variable initialized is 0, and poking around the state at this point is that Py_IsInitialized is "true" Calling the Py_Initialized(), which calls Py_InitializeEx(1). Next mod_python calls PyEval_InitThreads() which immediately returns because the python static variable interpreter_lock is not null. Next mod_python calls PyEval_SaveThread() which calls PyThreadState_Swap(NULL) which returns _PyThreadStateCurrent (which is NULL). PyEval_SaveThread() tests the return from the PyThreadState_Swap and this is where the fatal error happens. So.. it looks like Python thinks it's initialized, but it appears that the _PyThreadStateCurrent is NULL. (BTW -- sorry if I happen to state something obvious -- this is my first time through the mod_python internals, and I haven't done anything with invoking the python interpreter before..) |
|
Hrm... Could it be that something else initializes Python, e.g. do you have mod_wsgi loaded for example? |
|
I have a python ReWriteMap routine that is written in Python, and mod_passenger -- which introduced it's own python handler. I ran a test with mod_passenger disabled and things still failed. |
|
I disabled the ReWriteMap, and mod_passenger -- Still fails with same error |
|
Doing some more tracing, I put breakpoints at python_cleanup and python_cleanup_handler before invoking the apache reload. Neither of these break points hit. Is there supposed to be some code within mod_python that handles the "shutdown" request? I saw some code commented out with a references to MODPYTHON-109 and not invoking cleanup. |
|
I'd try disabling mod_python, restarting httpd and attaching to it with gdb and see if any of the Python symbols are loaded, e.g. setting a breakpoint on some CPython code, as well as double check that mod_python is indeed not loaded, just to eliminate that possibility. |
|
I removed all my references to mod_python (commented out the location that has the python handlers), attached gdb and did a breakpoint on PyEval_SaveThread. gdb asked if I wanted to set the breakpoint pending on a future shared library load. |
|
So we know the problem is that |
|
I put several break points and watches on. It looks like the lock isn't changing when the mod_python is reloaded. I wonder if the lock is being held across mod_python's module reload. I would think that mod_python should release any locks it owns when it's dropped, -or- it should remember that it has the lock across the module reload. |
|
Here is my theory (and I may have missed something in poking around in this) During apache graceful restart, all the modules get unloaded. It appears that the mod_python isn't called during the "unload" phase. The python interpreter is left loaded while the mod_python code has the interpreter lock. When apache reloads the modules, mod_python is loaded and python_init is invoked, and chugs its way through the initialization thinking that it's the first time. Python thinks it's already initialized and "assumes" that mod_python has the current lock. However mod_python thinks that it is initializing and assumes that it doesn't have the current lock. I would look at commit 4412c11 to see if it broke the graceful restart. Possible solutions? -- add code that captures the graceful restart, store what mod_python thingks is the current python state and test/restore it during python_init --or-- add logic that keeps the current state in persistent memory. Again... I'm not familiar enough with either the apache internals, mod_python, or the details of managing the python interpreter in a threaded environment, to be sure if my theory is even close to correct or if the solutions are even in the right ball park. |
|
Thank you, when I get a chance I'll take a look at this. |
|
Thank you. I'll be heading out for the evening. Let me know if there is anything you would like me to try/debug (though it will have to be in the morning) |
|
I'm also seeing this issue on Amazon Linux, using httpd 2.4.16, Python 2.7.10 and mod_python 3.5.0. During a restart now or a graceful restart, httpd will die, with the error previously noted: [Thu Nov 19 03:32:01.954307 2015] [mpm_prefork:notice] [pid 5885] AH00173: SIGHUP received. Attempting to restart It appears to happen on every "graceful restart" and "restart now" ("apachectl -k restart" (HUP) and "apachectl -k graceful" (USR1)). |
|
I can also confirm that this happens with mod_python 3.5.0 and the current git branch, but not with 3.4.1 - that stays alive. |
|
@robmoss2k thanks for the info, interesting, there seems to be an affinity with the latest Pythons. If I get a chance, I'll take a look at it over Thanksgiving, but PR's are always welcome! ;) |
|
@grisha Let me know if you would like something tested. I can recreate this at will and I can easily make updates for testing. |
|
I also have this problem on FreeBSD 10.2 x64 running apache 2.4, python 2.7.10, mod_python 3.5.0. Apache will run for a day or so but then crash with the following in the logs: I'm not so sure what the steps are for getting a backtrace but "gdb /usr/local/sbin/httpd /root/httpd.core" yields the following. Let me know if there's anything else that would be helpful. I've also posted this to FreeBSD Bugzilla. |
|
I've been able to replicate the issue: it happens with worker MPM after a graceful restart. Also by way of experiment tracked down that it starts happening with the Python release 2.7.9 (2.7.8 is ok). Haven't looked at what exactly causes it or how to fix it yet, may be later this weekend if I get some time. |
|
The Python commit that started causing this is: python/cpython@909877f and I have verified that simply |
|
I just ran a test, I "backed out" the above changes (I just edited /usr/lib64/python2.7/re.py and undid the _locale changes). I was able to do a graceful restart of apache without problems. I then went back and changed my modified re.py to just do an import on _locale and that change alone was enough to break the graceful reload |
|
The only thing that I can currently think is that python's _locale.so module is dynamically loaded after mod_python is loaded and because there is a reference, it's holding the python module in memory over the restart. |
|
I believe the latest check-in fixes this - could people try this in their set ups and confirm it? |
|
Initial testing looks good so far. Will update after further testing |
|
Sorry about the delay on getting back. The fix seems to have "fixed" the problem in my environment. |
|
Seems to be working now for me as well. Apache hasn't crashed since applying this fix. |
|
Six weeks of straight uptime. Looking good! |
|
I tested this adding this patch to the openSUSE package, and while it did fix the "apache crashes on reload" problem, it introduced a new one: I'm running ViewVC under mod_python, and these three lines are logged with every page load. ViewVC still appears to work fine, strangely. Notably, this path is missing all the site-package directories listed in sys.path in a CLI python session, including I tried adding this directory to the PythonPath in the apache config, but that had no effect. |
|
I noticed the same thing but it was happening for me before the patch. But, other than the extra lines in the log file, everything still seemed to work so I didn't give it much thought. |
|
I believe the initial PythonPath comes form libpython, so it's a function of how Python is configured, not mod_python (at least that's how I remember it). |
|
But you should also be able to change it via apache, no? Or is it stripping out anything it thinks is a "site" directory? I did not have these errors before the patch. This is how python is configured: The last four entries (all site-packages of some form) are missing from the path in the apache error log, as is the viewvc directory set below. Some directories are not present but that doesn't seem to matter. ViewVC apache configuration: I tried setting |
|
I had the same error |
|
@alecm3 I think you nailed it, I'll see about potentially fixing this, but until then it looks like it's a benign error. |
|
I had the same error I suggest the |
* Had to properly link in libpython2.7.so * Had to patch to fix grisha/mod_python#46 in python-2.7.16
|
this issue come back with Python 2.7.17 [Sun Mar 01 11:59:19.859132 2020] [mpm_event:notice] [pid 9500:tid 139896762713984] AH00489: Apache/2.4.41 (Unix) mod_python/3.5.0=none Python/2.7.17 mod_python was build from gitclone 2020-02-09 |
|
see: https://bugs.python.org/issue20891 I wasn't excited to make such stable in stable 2.7 and 3.6 anyway :-) |
|
apache handling musst be stop and start, restart give this error |
On apache httpd2 2.2.15 on CentOS release 6.5 (Final)
I am getting
mod_python: Creating 32 session mutexes based on 256 max processes and 0 max threads.
[Sun Sep 27 03:28:04 2015] [notice] mod_python: using mutex_directory /var/run/mod_python/
Fatal Python error: PyEval_SaveThread: NULL tstate
[Sun Sep 27 03:28:04 2015] [notice] seg fault or similar nasty error detected in the parent process
Is there any way to resolve it
The text was updated successfully, but these errors were encountered: