test_close_open_print_buffered(test_file) sometimes crashes #53541
Comments
|
I noticed test_close_open_print_buffered(test_file) sometimes crashes on release26-maint branch. This is most problamatic part in my python2.6 test survey on VC6 + Win2K. Here is stacktrace. _write_lk(int 6, const void * 0x00b49240, unsigned int 11) line 155 + 3 bytes Python crashes when MSVCRT touches internal buffer of file object via flush() in close(). I think this is multi-threading bug related to internal buffer of file object. I'll post more. |
ocean-city
mannequin
added
interpreter-core
|
I saw value of local variable *thread* was 5384 and internal buffer's thread(5384) func(file_close): enter.... Thread 5384 entered close_the_file() and ran local_close() and allowed |
|
I created a patch to workaround this crash. But I saw another problamatic phenomenon "Error without exception set". I confirmed PyErr_Occurred() returns TRUE when leaving file_close() on error, but I gave up. test_close_open_print_buffered (__main__.FileThreadingTests) ... close failed in
file object destructor:
IOError: [Errno 0] Error
Exception in thread Thread-1:
Traceback (most recent call last):
File "e:\python-dev\release26-maint\lib\threading.py", line 532, in __bootstra
p_inner
self.run()
File "e:\python-dev\release26-maint\lib\threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "test_custom.py", line 94, in worker
f()
File "test_custom.py", line 88, in <lambda>
lambda: self._close_and_reopen_file(),
File "test_custom.py", line 54, in _close_and_reopen_file
self._close_file()
File "test_custom.py", line 49, in _close_file
self.f.close()
SystemError: error return without exception setQclose failed in file object destructor: IOError: [Errno 0] Error
Exception in thread Thread-4:
Traceback (most recent call last):
File "e:\python-dev\release26-maint\lib\threading.py", line 532, in __bootstra
p_inner
self.run()
File "e:\python-dev\release26-maint\lib\threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "test_custom.py", line 94, in worker
f()
File "test_custom.py", line 88, in <lambda>
lambda: self._close_and_reopen_file(),
File "test_custom.py", line 54, in _close_and_reopen_file
self._close_file()
File "test_custom.py", line 49, in _close_file
self.f.close()
SystemError: error return without exception set4271 88.3166 ok ---------------------------------------------------------------------- OK |
|
The proposed resolution is wrong. As the comment says, there is a specific reason why f->fp is NULLed at this point. It appears that maybe MSVCRT is not entirely thread-safe with respect to buffered file I/O. Perhaps the test can simply be disabled under Windows. Is test_close_open_print_buffered the only test that fails? |
|
Maybe I am wrong, but when another thread calls File.close(), there is no system call on the code path, so I think still crash can happen nevertheless system call is thread safe. |
There is, and that's precisely what's the various tests test for. |
|
Hmm, correct me if following understanding is wrong somewhere...
Maybe can we fix this issue by the patch like this? I moved PyMem_Free Index: Objects/fileobject.c --- Objects/fileobject.c (revision 82910)
+++ Objects/fileobject.c (working copy)
@@ -371,9 +371,14 @@
Py_END_ALLOW_THREADS
if (sts == EOF)
return PyErr_SetFromErrno(PyExc_IOError);
- if (sts != 0)
+ if (sts != 0) {
+ PyMem_Free(f->f_setbuf);
+ f->f_setbuf = NULL;
return PyInt_FromLong((long)sts);
+ }
}
+ PyMem_Free(f->f_setbuf);
+ f->f_setbuf = NULL;
}
Py_RETURN_NONE;
}
@@ -567,12 +572,7 @@
static PyObject *
file_close(PyFileObject *f)
{
- PyObject *sts = close_the_file(f);
- if (sts) {
- PyMem_Free(f->f_setbuf);
- f->f_setbuf = NULL;
- }
- return sts;
+ return close_the_file(f);
} |
|
I forgot to mention that I didn't see any "Error without exception set" by my last patch. I agree f->f_fp = NULL; is needed there. ;-) |
I don't understand why you say that. Why is f->f_fp already NULL while By the way, is it 2.6-specific or does it also happen on 2.7? |
|
Does the following patch solve your issue? Index: Python/ceval.c --- Python/ceval.c (révision 82959)
+++ Python/ceval.c (copie de travail)
@@ -1763,6 +1763,8 @@
If __getattr__ raises an exception, w will
be freed, so we need to prevent that temporarily. */
Py_XINCREF(w);
+ if (PyFile_Check(w))
+ PyFile_IncUseCount((PyFileObject *) w);
if (w != NULL && PyFile_SoftSpace(w, 0))
err = PyFile_WriteString(" ", w);
if (err == 0)
@@ -1790,6 +1792,8 @@
else
PyFile_SoftSpace(w, 1);
}
+ if (PyFile_Check(w))
+ PyFile_DecUseCount((PyFileObject *) w);
Py_XDECREF(w);
Py_DECREF(v);
Py_XDECREF(stream); |
I tried, but it still crashed. I could reproduce the crash by following ////////////////////////////////////////////////////// import threading
import sys
if sys.version_info.major == 3:
xrange = range
def main():
size = 1024 * 1024 # large enough
f = open("__temp__.tmp", "w", size)
for _ in xrange(size):
f.write("c")
t1 = threading.Thread(target=f.close)
t2 = threading.Thread(target=f.close)
t1.start()
t2.start()
t1.join()
t2.join()
if __name__ == '__main__':
main()////////////////////////////////////////////////////// I lied a bit on msg110655. close(2) is not used, that is actually |
|
I cannot crash python by above patch anymore... Strange.
It seem to be. Is freeing the memory allocated for setvbuf() while close() running in another thread really safe? |
|
|
See the transient failures on "x86 XP-4" buildbot (2.6 and 2.7 only). Issue bpo-9555 closed as duplicate: |
I don't know. I guess nobody does that. |
|
I believe attached patch will fix this issue. Please forget |
|
Hirokazu, I've committed your 2.7 patch in r85892. Let's see what the buildbots say. |
|
Apparently it's fixed! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: