BUG: Python stops working when importing utide on Windows platform

[saulomeirelles]

Hi everyone,

I recently updated my Utide package on Windows 7 64-bit. Now every time I import utide, Python simply stops working and the program closes down. @ocefpaf has done some tests and confirmed the bug. I am using the following versions:

Scipy 0.17.0
Python 2.7.11
UTide 0.1

I ran my scripts on Linux and all went well.

Cheers, Saulo

[ocefpaf]

I narrowed it down to the from utide.utilities import loadmatbunch, so it might be a bug in scipy. I will look into it.

[ocefpaf]

I downgrading to scipy 0.16.0 fixes the problem on my WindowsXP-32 VM.

@efiring loadmatbunch is used in many libraries, did you get any report about it failing in other places with scipy >0.16.1?

[saulomeirelles]

Thanks, @ocefpaf ! That also worked for me.

[efiring]

https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/0OMcBlfWR_A might be the same thing.
@saulomeirelles, can you try using scipy 0.17 to read the ut_constants.mat file? If that fails, as I expect it will, then you could use that file as the test case in filing a bug report with scipy. This is quite serious.

[saulomeirelles]

Hi @efiring and @ocefpaf,

Scipy 0.17 failed to read ut_constants.mat on Windows platform.

Please refer to the attachment to check the log file and the script.

utide_test.zip

Traceback (most recent call last):
File "", line 2, in
tide_cnstit = spio.loadmat(mat_file[0])
File "C:\Users\Smeirellesnune\AppData\Local\Continuum\Miniconda\lib\site-packages\scipy\io\matlab\mio.py", line 134, in loadmat
MR = mat_reader_factory(file_name, appendmat, **kwargs)
File "C:\Users\Smeirellesnune\AppData\Local\Continuum\Miniconda\lib\site-packages\scipy\io\matlab\mio.py", line 58, in mat_reader_factory
mjv, mnv = get_matfile_version(byte_stream)
File "C:\Users\Smeirellesnune\AppData\Local\Continuum\Miniconda\lib\site-packages\scipy\io\matlab\miobase.py", line 241, in get_matfile_version
raise ValueError('Unknown mat file type, version %s, %s' % ret)
ValueError: Unknown mat file type, version 47, 47

[efiring]

@saulomeirelles, somehow you have gotten an html file in place of ut_constants.mat. The real ut_constants.mat is small:

efiring@manini2:~/work/programs/py/UTide/utide/data$ ll
total 264
-rw-rw-r--  1 efiring  efiring  124696 Jun  1  2015 FUV0.mat
-rw-rw-r--  1 efiring  efiring    6042 Feb 24  2015 ut_constants.mat

The file in your zipfile, which gives me the "version 47,47" error when I try to read it with scipy 0.16, is large:

efiring@manini2:~/temp$ unzip -l utide_test.zip utide_test/ut_constants.mat
Archive:  utide_test.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    45935  02-20-16 14:55   utide_test/ut_constants.mat
 --------                   -------
    45935                   1 file

and by opening it in a text editor you can verify that it is html.
It looks like you tried to download it from github, and you got the html download page instead of the file itself.

[saulomeirelles]

Ough! My bad! I'll check that asap.

Sent from my smartphone. Please forgive typos.
On Feb 20, 2016 5:02 PM, "Eric Firing" notifications@github.com wrote:

@saulomeirelles https://github.com/saulomeirelles, somehow you have
gotten an html file in place of ut_constants.mat. The real ut_constants.mat
is small:

efiring@manini2:~/work/programs/py/UTide/utide/data$ ll
total 264
-rw-rw-r-- 1 efiring efiring 124696 Jun 1 2015 FUV0.mat
-rw-rw-r-- 1 efiring efiring 6042 Feb 24 2015 ut_constants.mat

The file in your zipfile, which gives me the "version 47,47" error when I
try to read it with scipy 0.16, is large:

efiring@manini2:~/temp$ unzip -l utide_test.zip utide_test/ut_constants.mat
Archive: utide_test.zip
Length Date Time Name

---

45935  02-20-16 14:55   utide_test/ut_constants.mat

---

45935                   1 file

and by opening it in a text editor you can verify that it is html.
It looks like you tried to download it from github, and you got the html
download page instead of the file itself.

—
Reply to this email directly or view it on GitHub
#30 (comment).

[saulomeirelles]

Hi @efiring,

Python stopped working when loading ut_constant.mat. I did not even get the trace back message in the log file. You can find the script and other files in the attachment.

I've tested in different Scipy version. Apparently, Python crashes in all versions > 0.16.0.

utide_test.zip

[efiring]

@saulomeirelles Would you submit this as an issue to https://github.com/scipy/scipy/issues, please? Put the sample matfile in a gist (I think that is the right mechanism--@ocefpaf can advise) and link to that, as well as to the google groups message I noted in an earlier comment.

[efiring]

@saulomeirelles If you have time for some more testing, you could check some simpler matfiles to see if you can narrow down the problem. Is it the presence of unicode? Is it the nested structure? In any case, you can create the issue first and then add comments based on additional testing later. This is an important regression in scipy, but not one that will be immediately recognized by most scipy developers--like me, most probably don't use Windows.

[ocefpaf]

Saulo,

I am traveling (without a computer), but as, Eric said, that must be
reported. I can help you once I get back home.
On Feb 20, 2016 5:50 PM, "Eric Firing" notifications@github.com wrote:

@saulomeirelles https://github.com/saulomeirelles If you have time for
some more testing, you could check some simpler matfiles to see if you can
narrow down the problem. Is it the presence of unicode? Is it the nested
structure? In any case, you can create the issue first and then add
comments based on additional testing later. This is an important regression
in scipy, but not one that will be immediately recognized by most scipy
developers--like me, most probably don't use Windows.

—
Reply to this email directly or view it on GitHub
#30 (comment).

[ocefpaf]

@efiring Would you be OK to provide a npz version of the data instead? It seems that the issues opened (scipy/scipy#5882 and ContinuumIO/anaconda-issues#650) will remain unresolved for a while...

[efiring]

I think there can be python2/3 compatibility problems with npz. If those problems are not triggered by what we need to store here, then using npz is fine. Otherwise we could switch to netcdf and use the scipy netcdf reader. I could look into this in the next couple of days; but if you want to do it, that's fine too.

[ocefpaf]

I have a PR for npz almost ready. But I do like the netcdf idea.

[ocefpaf]

@saulomeirelles you can install the latest version using conda and the conda-forge community channel:

conda install -c conda-forge utide

That should work on Windows now.