Scipy test hangs (Trac #1724)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/1724 on 2012-08-25 by trac user adammiller, assigned to unknown.

Hello,

I am running scipy 0.10.1 on a Mac OS X 10.7.4. I am running Python 2.7.3 with Numpy 1.6.2. I installed scipy from source using a gfortran compiler. After importing scipy I ran

scipy.test(verbose=10)

to test that my installation was working properly. As far as I can tell all tests run fine until the code reaches a test that hangs. The output from scipy.test stops after the following:

make sure it handles relative values. ... ok

default behavior returns tempdir ... ok

make sure it handles relative values ... ok

Any ideas about why the test is hanging, or input on how I might fix this issue would be greatly appreciated.

[scipy-gitbot]

@rgommers wrote on 2012-08-25

It looks like it hangs inside weave. Can you confirm that by running

>>> from scipy import weave
>>> weave.test(verbose=2)

Can you please give the complete output of the above?

If that still hangs, see if this hangs too (extracted from weave/test_catalog.py TestDefaultDir):

from scipy import weave

path = weave.catalog.default_dir()
name = os.path.join(path,'dummy_catalog')
test_file = open(name,'w')
try:
    test_file.write('making sure default location is writable\n')
finally:
    test_file.close()
    os.remove(name)

[scipy-gitbot]

trac user adammiller wrote on 2012-08-26

Replying to [comment:1 rgommers]:

It looks like it hangs inside weave. Can you confirm that by running

>>> from scipy import weave
>>> weave.test(verbose=2)

Can you please give the complete output of the above?

That test hangs, Here is the output from that test.

>>> from scipy import weave
>>> weave.test(verbose=2)
Running unit tests for scipy.weave
NumPy version 1.6.2
NumPy is installed in /Library/Python/2.7/site-packages/numpy
SciPy version 0.10.1
SciPy is installed in /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy
Python version 2.7.3 (default, Aug 23 2012, 13:59:48) [GCC 4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.60))]
nose version 1.1.2
convert simple expr to blitz ... ok
convert fdtd equation to blitz. ... ok
convert simple expr to blitz ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior is to return current directory ... ok
make sure it handles relative values ... ok
test_simple (test_build_tools.TestConfigureSysArgv) ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior returns tempdir ... ok
make sure it handles relative values ... ok

If that still hangs, see if this hangs too (extracted from weave/test_catalog.py TestDefaultDir):

from scipy import weave

path = weave.catalog.default_dir()
name = os.path.join(path,'dummy_catalog')
test_file = open(name,'w')
try:
    test_file.write('making sure default location is writable\n')
finally:
    test_file.close()
    os.remove(name)

I am able to run this test without any hanging code.

[scipy-gitbot]

@rgommers wrote on 2012-08-26

Tests run in the same order for me, I paste here the output of waeve.test() with one line which shows the hanging test:

Python version 2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:52:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
nose version 1.1.2
convert simple expr to blitz ... ok
convert fdtd equation to blitz. ... ok
convert simple expr to blitz ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior is to return current directory ... ok
make sure it handles relative values ... ok
test_simple (test_build_tools.TestConfigureSysArgv) ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior returns tempdir ... ok
make sure it handles relative values ... ok
There should always be a writable file -- even if it is in temp ... ok

The last line comes from TestCatalog.test_access_writable_file() in weave/tests/test_catalog.py.
I don't really see anything obvious in catalog.py that would hang. Does this hang as well:

import os
from scipy.weave import catalog

q = catalog.catalog()
file = q.get_writable_file()
try:
    f = open(file,'w')
    f.write('bob')
finally:
    f.close()
    os.remove(file)

Since I can't reproduce this, it's difficult to debug. Do you know how to debug this?

[scipy-gitbot]

trac user adammiller wrote on 2012-08-28

Replying to [comment:3 rgommers]:

Tests run in the same order for me, I paste here the output of waeve.test() with one line which shows the hanging test:

Python version 2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:52:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
nose version 1.1.2
convert simple expr to blitz ... ok
convert fdtd equation to blitz. ... ok
convert simple expr to blitz ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior is to return current directory ... ok
make sure it handles relative values ... ok
test_simple (test_build_tools.TestConfigureSysArgv) ... ok
bad path should return same as default (and warn) ... ok
make sure it handles relative values. ... ok
default behavior returns tempdir ... ok
make sure it handles relative values ... ok
There should always be a writable file -- even if it is in temp ... ok

The last line comes from TestCatalog.test_access_writable_file() in weave/tests/test_catalog.py.
I don't really see anything obvious in catalog.py that would hang. Does this hang as well:

 import os
 from scipy.weave import catalog

 q = catalog.catalog()
 file = q.get_writable_file()
 try:
     f = open(file,'w')
     f.write('bob')
 finally:
     f.close()
     os.remove(file)

Since I can't reproduce this, it's difficult to debug. Do you know how to debug this?

I was able to run this without any issues.

>>> import os
>>> from scipy.weave import catalog
>>> q = catalog.catalog()
>>> file = q.get_writable_file()
>>> try:
...   f = open(file,'w')
...   f.write('bob')
... finally:
...   f.close()
...   os.remove(file)
... 
>>> 

I am unsure about how to otherwise debug the scipy test.

[scipy-gitbot]

@rgommers wrote on 2012-08-28

Hmm. You can run the test files one by one to narrow this down. Find it in site-packages, for example by using

from scipy import weave
print weave.__file__

it should be in weave/tests/test_catalog.py.

Then run it with python test_catalog.py; similar for other test files. Then you could comment out tests till you find the right one.

[scipy-gitbot]

@josef-pkt wrote on 2012-08-28

If you use nosetests, then you can run individual tests, for example:

>nosetests -v  C:\Python26\Lib\site-packages\scipy\weave\tests\test_catalog.py:TestCatalog.test_writable_dir

[scipy-gitbot]

trac user adammiller wrote on 2012-08-29

I have run each of the individual tests in the /site-packages/scipy/weave/tests/ directory. From there I find that test_c_spec.py and test_numpy_scalar_spec.py are the two tests that hang. Also, test_scxx_dict.py fails with a known error (I can provide the output if that would be helpful). For the two tests that stall I ran each individual test using nosetests. Each of these tests would hang individually. A copy of the individual tests that I ran is listed below:

nosetests -v test_c_spec.py:IntConverter.test_type_match_string
nosetests -v test_c_spec.py:IntConverter.test_type_match_int
nosetests -v test_c_spec.py:IntConverter.test_type_match_float
nosetests -v test_c_spec.py:IntConverter.test_type_match_complex
nosetests -v test_c_spec.py:IntConverter.test_var_in
nosetests -v test_c_spec.py:IntConverter.test_int_return
nosetests -v test_c_spec.py:FloatConverter.test_type_match_string
nosetests -v test_c_spec.py:FloatConverter.test_type_match_int
nosetests -v test_c_spec.py:FloatConverter.test_type_match_float
nosetests -v test_c_spec.py:FloatConverter.test_type_match_complex
nosetests -v test_c_spec.py:FloatConverter.test_float_var_in
nosetests -v test_c_spec.py:FloatConverter.test_float_return
nosetests -v test_c_spec.py:ComplexConverter.test_type_match_string
nosetests -v test_c_spec.py:ComplexConverter.test_type_match_int
nosetests -v test_c_spec.py:ComplexConverter.test_type_match_float
nosetests -v test_c_spec.py:ComplexConverter.test_type_match_complex
nosetests -v test_c_spec.py:ComplexConverter.test_complex_var_in
nosetests -v test_c_spec.py:ComplexConverter.test_complex_return
nosetests -v test_c_spec.py:FileConverter.test_py_to_file
nosetests -v test_c_spec.py:FileConverter.test_file_to_py
nosetests -v test_c_spec.py:CallableConverter.test_call_function
nosetests -v test_c_spec.py:SequenceConverter.test_convert_to_dict
nosetests -v test_c_spec.py:SequenceConverter.test_convert_to_list
nosetests -v test_c_spec.py:SequenceConverter.test_convert_to_string
nosetests -v test_c_spec.py:SequenceConverter.test_convert_to_tuple
nosetests -v test_c_spec.py:StringConverter.test_type_match_string
nosetests -v test_c_spec.py:StringConverter.test_type_match_int
nosetests -v test_c_spec.py:StringConverter.test_type_match_float
nosetests -v test_c_spec.py:StringConverter.test_type_match_complex
nosetests -v test_c_spec.py:StringConverter.test_var_in
nosetests -v test_c_spec.py:StringConverter.test_return
nosetests -v test_c_spec.py:ListConverter.test_type_match_bad
nosetests -v test_c_spec.py:ListConverter.test_type_match_good
nosetests -v test_c_spec.py:ListConverter.test_var_in
nosetests -v test_c_spec.py:ListConverter.test_return
nosetests -v test_c_spec.py:ListConverter.test_speed
nosetests -v test_c_spec.py:TupleConverter.test_type_match_bad
nosetests -v test_c_spec.py:TupleConverter.test_type_match_good
nosetests -v test_c_spec.py:TupleConverter.test_var_in
nosetests -v test_c_spec.py:TupleConverter.test_return
nosetests -v test_c_spec.py:DictConverter.test_type_match_bad
nosetests -v test_c_spec.py:DictConverter.test_type_match_good
nosetests -v test_c_spec.py:DictConverter.test_var_in
nosetests -v test_c_spec.py:DictConverter.test_return

nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.setUp
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_type_match_string
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_type_match_int
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_type_match_float
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_type_match_complex128
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_complex_var_in
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_complex_return
nosetests -v test_numpy_scalar_spec.py:NumpyComplexScalarConverter.test_inline

[scipy-gitbot]

@rgommers wrote on 2012-09-08

Thanks Adam. I'm really puzzled by this. No one else has reported a similar issue on OS X 10.7. That could be because those tests are not normally run (except with scipy.test('full') or nosetests, they're all decorated as slow). Or because there's something about your setup or C++ compiler that's unusual.

Not sure what to do about this now. In a month or two I should have an OS X >10.6 system to try and reproduce this.

[person142]

Since this is a weave issue (and weave is no longer a part of SciPy) I added it to scipy/weave#7 and am closing it here.