arpack tests fail when installed on mac os x

[joernhees]

I ran into this problem several times already with different installation methods. This time i used brew to install scipy:

brew tap homebrew/science
brew tap samueljohn/python
brew update
brew install python
pip install nose
brew install scipy

If you then run the tests the following errors occur: https://gist.github.com/joernhees/5733199

A shortened version here:

$ ipython
Python 2.7.5 (default, Jun 8 2013, 00:27:06)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

In [1]: import scipy

In [2]: scipy.test()
Running unit tests for scipy
NumPy version 1.7.1
NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
SciPy version 0.12.0
SciPy is installed in /usr/local/lib/python2.7/site-packages/scipy
Python version 2.7.5 (default, Jun 8 2013, 00:27:06) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]
nose version 1.3.0
/usr/local/lib/python2.7/site-packages/numpy/lib/utils.py:139: DeprecationWarning: `scipy.lib.blas` is deprecated, use `scipy.linalg.blas` instead!
warnings.warn(depdoc, DeprecationWarning)
/usr/local/lib/python2.7/site-packages/numpy/lib/utils.py:139: DeprecationWarning: `scipy.lib.lapack` is deprecated, use `scipy.linalg.lapack` instead!
warnings.warn(depdoc, DeprecationWarning)
...............................................K...............................................K..K........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSSSSS......SSSSSS......S..........K..............................................................................................................................................................................................................................................................................................K...................................................................................................................................................................................................../usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
/usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
/usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
/usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
/usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
./usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
./usr/local/lib/python2.7/site-packages/scipy/misc/pilutil.py:230: DeprecationWarning: fromstring() is deprecated. Please call frombytes() instead.
image = Image.fromstring('L',shape,bytedata.tostring())
.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................FFF.....FFF..FF......FF..F..........FFF.....FFF..............FF..FF.FFF..FF..FF..FF..F..........FF..FFF.FFF.............F.......F....F.......F..............FFF......FF.............F...F...F....F...F...F..............FFF.FFF.FFF..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................................K......................................................................................................................................................................KK.................................................................................................................................................................................................................................................................................................................................................................................................................................K........K.......................................................................................................................................................................................................................S...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
FAIL: test_arpack.test_symmetric_modes(True, <std-symmetric>, 'f', 2, 'LM', None, 0.5, <class 'scipy.sparse.csr.csr_matrix'>, None, 'normal')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 259, in eval_evec
assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1179, in assert_allclose
verbose=verbose, header=header)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 645, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.00178814, atol=0.000357628
error for eigsh:standard, typ=f, which=LM, sigma=0.5, mattype=csr_matrix, OPpart=None, mode=normal
(mismatch 100.0%)
x: array([[ 2.38156418e-01, -6.75444982e+09],
[ -1.07853470e-01, -8.01245676e+09],
[ 1.24683023e-01, -5.19757686e+09],...
y: array([[ 2.38156418e-01, -5.70949789e+08],
[ -1.07853470e-01, -4.05829392e+08],
[ 1.24683023e-01, 6.25800146e+07],...

======================================================================
FAIL: test_arpack.test_symmetric_modes(True, <std-symmetric>, 'f', 2, 'LM', None, 0.5, <class 'scipy.sparse.csr.csr_matrix'>, None, 'buckling')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 259, in eval_evec
assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1179, in assert_allclose
verbose=verbose, header=header)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 645, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.00178814, atol=0.000357628
error for eigsh:standard, typ=f, which=LM, sigma=0.5, mattype=csr_matrix, OPpart=None, mode=buckling
(mismatch 100.0%)
x: array([[ 3.53755447e-01, -2.29114355e+04],
[ -1.60204595e-01, -6.65625445e+04],
[ 1.85203065e-01, -2.69012500e+04],...
y: array([[ 3.53755447e-01, -8.88255444e+05],
[ -1.60204595e-01, -2.39343354e+06],
[ 1.85203065e-01, -3.96842525e+04],...

======================================================================
FAIL: test_arpack.test_symmetric_modes(True, <std-symmetric>, 'f', 2, 'LM', None, 0.5, <class 'scipy.sparse.csr.csr_matrix'>, None, 'cayley')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 259, in eval_evec
assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1179, in assert_allclose
verbose=verbose, header=header)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 645, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.00178814, atol=0.000357628
error for eigsh:standard, typ=f, which=LM, sigma=0.5, mattype=csr_matrix, OPpart=None, mode=cayley
(mismatch 100.0%)
x: array([[ -2.38156418e-01, 1.04661597e+09],
[ 1.07853470e-01, 1.39930271e+09],
[ -1.24683023e-01, 9.56906461e+08],...
y: array([[ -2.38156418e-01, 7.63721281e+07],
[ 1.07853470e-01, 1.25169905e+08],
[ -1.24683023e-01, 2.91283130e+07],...

======================================================================
...
======================================================================
FAIL: test_arpack.test_symmetric_modes(True, <gen-symmetric>, 'd', 2, 'SA', None, 0.5, <function asarray at 0x10ad03e60>, None, 'cayley')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 259, in eval_evec
assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1179, in assert_allclose
verbose=verbose, header=header)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 645, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=4.44089e-13, atol=4.44089e-13
error for eigsh:general, typ=d, which=SA, sigma=0.5, mattype=asarray, OPpart=None, mode=cayley
(mismatch 100.0%)
x: array([[-0.36892684, -0.01935691],
[-0.26850996, -0.11053158],
[-0.40976156, -0.13223572],...
y: array([[-0.43633077, -0.01935691],
[-0.25161386, -0.11053158],
[-0.36756684, -0.13223572],...

----------------------------------------------------------------------
Ran 4831 tests in 54.169s

FAILED (KNOWNFAIL=11, SKIP=14, failures=63)
Out[2]: <nose.result.TextTestResult run=4831 errors=0 failures=63>

In [3]:

[joernhees]

I actually found a workaround by using openblas (a bit slower is better than wrong IMO):

First make sure you have a gfortran version >= 4.7: gfortran --version (If not you probably followed some popular mac os x scipy install howto before. In my case it used brew's apple-gcc42, so brew unlink apple-gcc42 && brew install gfortran could solve that).

Then run:

brew install openblas
brew reinstall scipy --with-openblas

As mentioned in samueljohn/homebrew-python#12 no one wants to be responsible for these errors...
IMHO scipy should at least warn users in case a library is used which is known to cause errors of this magnitude.

[rgommers]

It's not "doesn't want to be responsible", it's simply quite hard to solve. Duplicate of gh-2372, so closing.

[joernhees]

@rgommers i'm quite sure this isn't a dup of that, i'm running on mac os x and afaik it uses Accelerate.framework in LAPACK.

Also i seem to judge the severity of this a bit different: Many of my researcher colleagues use macs and all of whom I asked to run the scipy test were surprised that these tests fail. I'm not 100 % sure how bad the errors actually are, but i get a bad feeling that many scientific publications could depend on this and never notice the errors.

What i suggest is not hard to solve: i suggested that scipy should check and notify the users if the wrong library is used which is known to cause problems. You point out yourself that not using Accelerate and llvm is the best in your opinion ( samueljohn/homebrew-python#12 ). Why not check for it?

[pv]

Yes, disabling support for Accelerate completely could be sensible until this is resolved.

[rgommers]

Sorry about linking the wrong ticket. It is a duplicate of gh-2256 as well as gh-2248.

I'm not sure about disabling Accelerate support - it will create a lot of build issues and a flood of ticket / complaints on the mailing list. Disabling this part of ARPACK functionality (everywhere or only on OS X?) is probably a better alternative. AFAIK it's only single precision that has a problem and that's not widely used.

I suggest continuing this discussion on gh-2248; I'll add that to the 0.13.0 Milestone so we can't just forget about this. Closing again as duplicate.

[pv]

@rgommers: I meant that we disable also Veclib/Accelerate detection. This is certainly doable.

The problem is that the "part of ARPACK functionality" in question is not precisely defined. It's not even only single precision, also apparently double precision fails (see 'd' in the above tests).

[rgommers]

I understood that that's what you meant. It's doable, but will create lots of issues for users. We'll be disabling the only BLAS/LAPACK shipped with the OS, forcing users to hunt for other binaries (which may have their own issues) or install a BLAS/LAPACK from source (very much nontrivial). So it's not a decision to take lightly.

OpenBLAS support in numpy.distutils is still shaky by the way: http://thread.gmane.org/gmane.comp.python.numeric.general/53541/focus=53547

[rgommers]

You're right by the way, missed the 'd'.

[rgommers]

It's well-defined enough - disable everything that's failing on OS X 10.8 right now.

[pv]

I think disabling is not a satisfactory solution, because: the tests show that something fails, we do not understand why. However, they do not show that the parts for which the tests pass function correctly, rather, they show they work in some test cases.

It will also break portability of Scipy code across platforms.

Moreover, the situation has apparently got worse with new OSX versions, as previously it used to be so that the problems were restricted to single precision. Next version of OSX --- more problems?

[rgommers]

Could be that OS X 10.9 has even more issues, we'll find out soon enough.

There's also the -ff2c solution still, gh-280. On that same PR David says that MKL has similar issues as Accelerate. OpenBLAS and GotoBLAS are not well supported by numpy.distutils. Bento support is better, but has 64-bit issues. So what are we going to recommend, use Netlib BLAS/LAPACK or ATLAS only?

I'm going to try one more time: all test failures are in the generalized-symmetric support, as added in gh-25. It was added for 0.10.0, and started given issues soon after. Disabling generalized-symmetric support (a feature used by a tiny fraction of users) either on OS X or on all platforms would be a lower-impact change than completely removing support for Accelerate.

@jakevdp @cournape any opinion on this? Time to have a go at fixing it? I believe you both have an OS X 10.8 system:)