io tests crash on Windows #5540

ev-br · 2015-11-25T17:26:45Z

test_mio.test_load('double', ['C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_4.2c_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.1_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.5.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.4_GLNX86.mat'], {'testdouble': array([[ 0.        ,  0.78539816,  1.57079633,  2.35619449,  3.14159265, ...

Windows XP, numpy-vendor binaries.

The text was updated successfully, but these errors were encountered:

rgommers · 2015-11-25T22:04:57Z

Hmm, very little changed in io.matlab for this release. Do the 0.16.1 binaries crash as well?

ev-br · 2015-11-26T09:05:20Z

io tests pass for 0.16.1 binaries from github releases.
With binaries for yesterday's master, I tracked this down to https://github.com/scipy/scipy/blob/master/scipy/io/matlab/mio.py#L135 which seems to suggest #5082
@matthew-brett @anntzer any ideas?

matthew-brett · 2015-11-26T19:44:05Z

@ev-br - can you post a more complete log? Would it be easy to post a binary somewhere for me to test?

ev-br · 2015-11-26T20:17:17Z

I've posted the binary at https://github.com/scipy/scipy/releases/tag/v0.17pre
(This seemed to "create a release", whatever is meant by Github; I'm not sure what sort of publicity it entails; if that's undesirable, I'll immediately remove it and send you a dropbox link or something.)

ev-br · 2015-11-26T20:19:53Z

Regarding a more complete log, I'm not sure what it would be. I just ran scipy.test(verbose=10) and then walked it a bit in pdb. What sort of info would be useful for you?

matthew-brett · 2015-11-26T20:26:26Z

I was thinking of something like the complete log from nosetests /path/to/scipy/install/scipy/io/matlab/tests/test_mio.py:test_load

ev-br · 2015-11-26T20:56:00Z

Under Wine this apparently just hangs:

vagrant@vagrant-ubuntu-precise-32:~$ .wine/drive_c/Python27/Scripts/nosetests.exe .wine/drive_c/Python27/Lib/site-packages/scipy/io/matlab/renamed/test_mio.py:test_load
^Cerr:console:GenerateConsoleCtrlEvent Invalid event 46 for PGID 0
^Cerr:console:GenerateConsoleCtrlEvent Invalid event 46 for PGID 0
^Cerr:console:GenerateConsoleCtrlEvent Invalid event 46 for PGID 0
^Cerr:console:GenerateConsoleCtrlEvent Invalid event 46 for PGID 0

In a different window:

vagrant@vagrant-ubuntu-precise-32:~$ ps ax | grep nosetests
 2828 pts/0    S+     0:00 .wine/drive_c/Python27/Scripts/nosetests.exe .wine/drive_c/Python27/Lib/site-packages/scipy/io/matlab/renamed/test_mio.py:test_load                                      
 2831 pts/0    S+     2:40 C:\Python27\python.exe Z:\home\vagrant\.wine\drive_c\Python27\Scripts\nosetests-script.py .wine/drive_c/Python27/Lib/site-packages/scipy/io/matlab/renamed/test_mio.py:test_load

only reacts to kill -9.

ev-br · 2015-11-26T20:57:11Z

Under WIndows it crashes, I'll be able to post the exact details in the morning local time.

matthew-brett · 2015-11-26T20:59:02Z

Just to be sure - if you build scipy 0.16.1 with the same setup, you get no such crash, and all tests pass?

ev-br · 2015-11-26T22:48:17Z

OK, something's not right with my build setup. If I take official superpack installers, all io tests pass.

If I build 0.16.1, I get the same crash:

>>> io.test(verbose=10)
Running unit tests for scipy.io
NumPy version 1.7.2
NumPy is installed in C:\Python27\lib\site-packages\numpy
SciPy version 0.16.1
SciPy is installed in C:\Python27\lib\site-packages\scipy
Python version 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]
nose version 1.3.7
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext']
test1 (test_arffread.DataTest) ... ok
test2 (test_arffread.DataTest) ... ok
test3 (test_arffread.DataTest) ... ok
test_filelike (test_arffread.DataTest) ... ok
test_date_attribute (test_arffread.DateAttributeTest) ... ok
test_datetime_local_attribute (test_arffread.DateAttributeTest) ... ok
test_datetime_missing (test_arffread.DateAttributeTest) ... ok
test_datetime_timezone (test_arffread.DateAttributeTest) ... ok
test_month_attribute (test_arffread.DateAttributeTest) ... ok
test_year_attribute (test_arffread.DateAttributeTest) ... ok
test_badtype_parsing (test_arffread.HeaderTest) ... ok
test_dateheader (test_arffread.HeaderTest) ... ok
test_dateheader_unsupported (test_arffread.HeaderTest) ... ok
test_fullheader1 (test_arffread.HeaderTest) ... ok
test_type_parsing (test_arffread.HeaderTest) ... ok
test_missing (test_arffread.MissingDataTest) ... ok
test_from_number (test_fortran_format.TestExpFormat) ... ok
test_to_fortran (test_fortran_format.TestExpFormat) ... ok
test_exp_exp (test_fortran_format.TestFortranFormatParser) ... ok
test_repeat_exp (test_fortran_format.TestFortranFormatParser) ... ok
test_repeat_exp_exp (test_fortran_format.TestFortranFormatParser) ... ok
test_simple_exp (test_fortran_format.TestFortranFormatParser) ... ok
test_simple_int (test_fortran_format.TestFortranFormatParser) ... ok
test_simple_repeated_int (test_fortran_format.TestFortranFormatParser) ... ok
test_wrong_formats (test_fortran_format.TestFortranFormatParser) ... ok
test_from_number (test_fortran_format.TestIntFormat) ... ok
test_to_fortran (test_fortran_format.TestIntFormat) ... ok
test_simple (test_hb.TestHBReader) ... ok
test_simple (test_hb.TestRBRoundtrip) ... ok
test_byteordercodes.test_native ... ok
test_byteordercodes.test_to_numpy ... ok
test_mio.test_load('double', ['C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_4.2c_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.1_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.5.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.4_GLNX86.mat'], {'testdouble': array([[ 0.        ,  0.78539816,  1.57079633,  2.35619449,  3.14159265, ...

... "pythonw.exe encountered a problem and needs to close."

Hmm...
What does the binary from up thread do on your machine Matthew?

matthew-brett · 2015-11-27T00:30:21Z

Sorry I am away from my computer doing Thanksgiving stuff this evening will check tomorrow

ev-br · 2015-11-27T07:13:12Z

Happy Thanksgiving!

matthew-brett · 2015-11-28T00:02:02Z

I installed your installer, using the numpy superpack installer for numpy. I also get a crash for test_mio.py.

The scipy 0.16.1 installer downloaded from SF also works fine for me.

ev-br · 2015-11-29T17:13:52Z

OK, I tried building scipy versions down to 0.15.1 and with cython 0.23.4 or 0.22, and they all crash. Now, this seems to point to a problem in my setup, but it's just an unmodified numpy/numpy-vendor#10. At this stage, I'd appreciate any help or any sort of pointers.

rgommers · 2015-11-30T08:00:54Z

I'll try to reproduce tonight.

rgommers · 2015-12-01T06:24:28Z

Hmm, reproduced, but not yet clear what changed. The only thing I can think of right now is the switch from Cython 0.20.0 to 0.22.0 and then 0.23.4.

ev-br · 2015-12-01T10:02:53Z

OK, just as a data point: I downgraded to Cython 0.20.0 and build scipy 0.15.1 with it. (I don't think later scipy versions could be built with it, can they). Under Wine, the problem shows up as a deadlock.

>>> import scipy.io.matlab as iom
>>> iom.test(verbose=10)
Running unit tests for scipy.io.matlab
NumPy version 1.6.2
NumPy is installed in C:\Python27\lib\site-packages\numpy
SciPy version 0.15.1
SciPy is installed in C:\Python27\lib\site-packages\scipy
Python version 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]
nose version 1.1.2
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext']
nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext']
test_byteordercodes.test_native ... ok
test_byteordercodes.test_to_numpy ... ok
test_mio.test_load('double', ['C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.5.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_6.1_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.1_GLNX86.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_4.2c_SOL2.mat', 'C:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testdouble_7.4_GLNX86.mat'], {'testdouble': array([[ 0.        ,  0.78539816,  1.57079633,  2.35619449,  3.14159265, ... err:ntdll:RtlpWaitForCriticalSection section 0x785b7428 "?" wait timed out in thread 0009, blocked by 0000, retrying (60 sec)
err:ntdll:RtlpWaitForCriticalSection section 0x785b7428 "?" wait timed out in thread 0009, blocked by 0000, retrying (60 sec)
err:ntdll:RtlpWaitForCriticalSection section 0x785b7428 "?" wait timed out in thread 0009, blocked by 0000, retrying (60 sec)

Not sure if this makes it any clearer...

rgommers · 2015-12-01T21:49:13Z

I think the issue is with np.fromfile here. Can reproduce it also with cluster/test_vq.py which has fromfile uses. I'll see if I can prove and work around it.

This change is faster (avoids multiple times reading of the same file), and may avoid a hang of the test suite under Wine (see scipygh-5540).

rgommers · 2015-12-01T22:35:03Z

https://github.com/rgommers/scipy/tree/remove-fromfile fixes one crash in cluster. I misread test_mmio for test_mio. io/matlab doesn't contain a call to fromfile. So will have to dig a bit deeper for that one. Good chance it's related though, symptoms are identical.

rgommers · 2015-12-02T23:21:14Z

OK, I see the same thing in the numpy tests and in other tests that use np.load and np.fromfile at least. If I download a numpy 1.6.2 superpack from SF and install that, the numpy issues and all scipy issues except the test_mio.test_load one are gone. The 1.6.2 binaries on SF were built with an older Wine version, that's the only difference AFAIK.

So the good news is that we narrowed it down a bit, the bad news that this is still not going to be fun.

rgommers · 2015-12-02T23:33:04Z

From vague memories: there are issues with file handles and fseek/ftell. The io.matlab code that looks suspicious here is the use of npy_PyFile_Dup and npy_PyFile_DupClose. Those have been removed from numpy in 1.10.0, but are still used in streams.pyx. These functions seem to be copied from numpy into io/matlab/py3k.h. That file was never touched after initial creation when we first started supporting Python 3.x.

rgommers · 2015-12-02T23:42:27Z

Just checked: np.fromfile uses npy_PyFile_DupClose2 and np.load uses fromfile under the hood - all the same issue it looks like.

rgommers · 2015-12-02T23:53:32Z

History:
numpy/numpy#2180
numpy/numpy#4118
numpy/numpy#4152

This change is faster (avoids multiple times reading of the same file), gets rid of a data file that's not really needed, and may avoid issues with np.fromfile (under Wine for example, see scipygh-5540).

ev-br · 2015-12-17T11:54:05Z

It's not fun indeed, but it seems a bit different of an issue here. The numpy issues relate to python 3, but here python 3 is actually fine, the crash is on python 2.7.

On python 2, npy_PyFile_Dup et al are just aliases to functions from Python.h.
I tried to avoid using them and use the numpy replacements on python 2: master...ev-br:io_crash
And it does not crash --- there are "just" 70-odd copies of

======================================================================
ERROR: test_mio.test_read_both_endian
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "Z:\home\vagrant\.wine\drive_c\Python27\Lib\site-packages\scipy\io\matlab\tests\test_mio.py", line 904, in test_read_both_endian
    d = rdr.get_variables()
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 268, in get_variables
    self.initialize_read()
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 196, in initialize_read
    self._file_reader = VarReader5(self)
  File "scipy/io/matlab/mio5_utils.pyx", line 191, in scipy.io.matlab.mio5_utils.VarReader5.__cinit__ (scipy\io\matlab\mio5_utils.c:2846)
  File "scipy/io/matlab/mio5_utils.pyx", line 214, in scipy.io.matlab.mio5_utils.VarReader5.set_stream (scipy\io\matlab\mio5_utils.c:3358)
  File "scipy/io/matlab/streams.pyx", line 360, in scipy.io.matlab.streams.make_stream (scipy\io\matlab\streams.c:5543)
SystemError: NULL result without error in PyObject_Call

Full log: https://gist.github.com/ev-br/63a0473bcc84820190a1

This seems to point to some CRT incompatibility? https://mail.scipy.org/pipermail/numpy-discussion/2012-July/063414.html

pv · 2015-12-17T14:41:24Z

Some issues:
.

The Python 2 code is indeed obtaining FILE* handles from the CRT
Python uses and passing them to whatever CRT Scipy uses. This problem
probably manifests already with numpy.fromfile, built in same way?
.
The call to *DupClose in streams.pyx:FileStream.__del__ is not
guarded for the case where initialization in __init__ fails.
.
The routines in py3k.h should be synced with Numpy. However, I don't
think this is related to this crash, since on Python 2 it's just
#defines for the Python C-API.
.
If the issue is some CRT issue, then I think one should avoid adding
workarounds, and try to understand why the build environment produces
incorrect binaries.

rgommers · 2015-12-23T06:12:08Z

For the record, my ancient Wine 1.1.39 setup doesn't have this issue. Something got messed up in numpy-vendor.

rgommers · 2016-01-03T19:47:51Z

We're not producing win32 binaries for 0.17.0, so removing the 0.17.0 milestone.

Leaving the issue open because there's some things to do - see comment @pv above.

pv · 2016-02-22T17:11:38Z

Maybe related: #5882 (Anaconda binaries)

sethtroisi · 2020-05-13T21:39:09Z

I believe that all of the mentioned issues above have been resolved.

Python2 no longer supported
DupClose in streams.pyx removed in favor of GenericStream
py3k.h all removed (and numpy_py3k.h almost completely gone)
windows 32 binaries gone

ev-br added scipy.io defect A clear bug or issue that prevents SciPy from being installed or used as expected labels Nov 25, 2015

ev-br added this to the 0.17.0 milestone Nov 25, 2015

This was referenced Nov 26, 2015

NetCDF maskandscale test failure on Windows XP #5541

Closed

and some more test failures #5542

Closed

rgommers added a commit to rgommers/scipy that referenced this issue Dec 1, 2015

TST: remove use of np.fromfile from cluster.vq tests.

92c38ec

This change is faster (avoids multiple times reading of the same file), and may avoid a hang of the test suite under Wine (see scipygh-5540).

rgommers mentioned this issue Dec 3, 2015

TST: remove use of np.fromfile from cluster.vq tests #5573

Merged

rgommers removed this from the 0.17.0 milestone Jan 3, 2016

sethtroisi closed this as completed May 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

io tests crash on Windows #5540

io tests crash on Windows #5540

ev-br commented Nov 25, 2015

rgommers commented Nov 25, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 27, 2015 via email

ev-br commented Nov 27, 2015

matthew-brett commented Nov 28, 2015

ev-br commented Nov 29, 2015

rgommers commented Nov 30, 2015

rgommers commented Dec 1, 2015

ev-br commented Dec 1, 2015

rgommers commented Dec 1, 2015

rgommers commented Dec 1, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

ev-br commented Dec 17, 2015

pv commented Dec 17, 2015

rgommers commented Dec 23, 2015

rgommers commented Jan 3, 2016

pv commented Feb 22, 2016

sethtroisi commented May 13, 2020

io tests crash on Windows #5540

io tests crash on Windows #5540

Comments

ev-br commented Nov 25, 2015

rgommers commented Nov 25, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 26, 2015

ev-br commented Nov 26, 2015

matthew-brett commented Nov 27, 2015 via email

ev-br commented Nov 27, 2015

matthew-brett commented Nov 28, 2015

ev-br commented Nov 29, 2015

rgommers commented Nov 30, 2015

rgommers commented Dec 1, 2015

ev-br commented Dec 1, 2015

rgommers commented Dec 1, 2015

rgommers commented Dec 1, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

rgommers commented Dec 2, 2015

ev-br commented Dec 17, 2015

pv commented Dec 17, 2015

rgommers commented Dec 23, 2015

rgommers commented Jan 3, 2016

pv commented Feb 22, 2016

sethtroisi commented May 13, 2020