Errors in tests of Scipy #12

Closed
dmitry-kabanov opened this Issue Feb 3, 2013 · 6 comments

Projects

None yet

4 participants

@dmitry-kabanov

After installation of scipy I executed scipy tests:

$ python
Python 2.7.3 (default, Feb  2 2013, 19:22:32) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy
>>> scipy.test()

After tests have run I got a lot of failures:

Ran 4230 tests in 55.406s

FAILED (KNOWNFAIL=10, SKIP=27, failures=65)
<nose.result.TextTestResult run=4230 errors=0 failures=65>

All errors (I suppose) are connected with test_arpack.py:

FAIL: test_arpack.test_symmetric_modes(True, <gen-symmetric>, 'd', 2, 'SA', None, 0.5, <function asarray at 0x10cb09de8>, None, 'cayley')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 257, in eval_evec
    assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=4.44089e-13, atol=4.44089e-13
error for eigsh:general, typ=d, which=SA, sigma=0.5, mattype=asarray, OPpart=None, mode=cayley
(mismatch 100.0%)
 x: array([[-0.36892684, -0.01935691],
       [-0.26850996, -0.11053158],
       [-0.40976156, -0.13223572],...
 y: array([[-0.43633077, -0.01935691],
       [-0.25161386, -0.11053158],
       [-0.36756684, -0.13223572],...

Is it OK that I have so many errors? Are these errors known? Or is it just only my problem?

@samueljohn samueljohn was assigned Feb 3, 2013
@samueljohn
Contributor

I guess this is the thread about these errors:
http://thread.gmane.org/gmane.comp.python.scientific.devel/15289/focus=15297

They are known with Accelerate.framework and I can't do anything about it personally. It's a SciPy thing.

Perhaps see also here:
http://projects.scipy.org/scipy/search?q=os+x+arpack&noquickjump=1&ticket=on&changeset=on&milestone=on&wiki=on&wiki=on

As I understand these are basically corner-case sign-handling stuff for sparse arrays.

@samueljohn
Contributor

I think particularly this ticket http://projects.scipy.org/scipy/ticket/1729

@rgommers writes:

No good workaround, the final solution should be to simply not use the Accelerate Framework anymore, but standard ATLAS.

All failures are for single-precision, so likely won't affect you.

So basically, you can use --with-openblas to build my numpy/scipy formulae (please rebuilt both) and I have to investigate how to link OpenBLAS against a multi-threaded and optimized LAPACK. Because OpenBLAS has good BLAS performance but the linear algebra routines (LAPACK) are basically vanilla and slower than Accelerate.framework.

@rgommers any other suggestions for Mac Users? Shall we stop to build against Accelerate.framework? Sad thing is that Accelerate.framework is really fast on Macs.

@rgommers
rgommers commented Feb 4, 2013

That seems to be the best solution. Not using Accelerate (and llvm-gcc) at all is the best thing to do imho.

@ccwang002

I tried to install Numpy 1.7.1 and Scipy 0.12.0 on OSX 10.7 (Lion) and got into same test failures for Scipy. After some surveys, I managed to pass all numpy and scipy tests on both Python 2.7.5 and 3.3.2 based on the note by jonathansick. Whole LAPACK and ATLAS can now be built successfully on mac, so we may not need to depend on the Accelerate framework.

Here is my installation script, which builds ATLAS 3.10.0 and LAPACK 3.4.2.

$ brew install gfortran
$ brew install python
$ brew install suite-sparse  # dependency for numpy
$ brew install pcre swig  # dependency for scipy

# install ATLAS and LAPACK
$ cd /tmp
$ wget http://sourceforge.net/projects/math-atlas/files/Stable/3.10.0/atlas3.10.0.tar.bz2
$ wget http://www.netlib.org/lapack/lapack-3.4.2.tgz
$ bunzip2 -c atlas3.10.0.tar.bz2 | tar xfm -
$ mv ATLAS ATLAS3.10.0
$ cd ATLAS3.10.0
$ mkdir _mac_build   # cannot be built at source dir
$ cd _mac_build
# by --with-netlib-lapack-tarfile, inculde full LAPACK as well
$ ../configure -b 64 --prefix=/usr/local/atlas --shared --with-netlib-lapack-tarfile=/tmp/lapack-3.4.2.tgz  
$ make build
$ make check
$ make ptcheck
$ make time
$ make install

# install numpy and scipy, following code in their current receipts
$ export ATLAS=/usr/local/atlas   # the path to built ATLAS and LAPACK

$ cd /tmp
$ wget https://pypi.python.org/packages/source/n/numpy/numpy-1.7.1.tar.gz
$ tar zxvf numpy-1.7.1.tar.gz
$ cd numpy-1.7.1
$ python2 setup.py build --fcompiler=gnu95 install

$ cd /tmp
$ wget https://pypi.python.org/packages/source/s/scipy/scipy-0.12.0.tar.gz
$ tar zxvf scipy-0.12.0.tar.gz
$ cd scipy-0.12.0
$ python2 setup.py build --fcompiler=gnu95 install

I passed all tests for numpy and scipy on Python 2.7.5.

$ python2 -c 'import numpy as n; n.test("full")'
Ran 4809 tests in 48.978s
OK (KNOWNFAIL=5, SKIP=5)

$ python2 -c 'import scipy as s; s.test()'
Ran 6134 tests in 77.866s
OK (KNOWNFAIL=15, SKIP=44)

On Python 3.3.2 I follow the same procedure by replacing python2 with python3, and then build numpy and scipy again. I passed all tests except for one fail in scipy since weave only supports Python 2.X.

$ python3 -c 'import numpy as n; n.test("full")'
Ran 4808 tests in 56.670s
OK (KNOWNFAIL=6, SKIP=5)

$ python3 -c 'import scipy as s; s.test()'
ERROR: Failure: ImportError (scipy.weave only supports Python 2.x)
Ran 5999 tests in 136.962s
FAILED (KNOWNFAIL=16, SKIP=44, errors=1)

Maybe we can add ATLAS and LAPACK as a new Homebrew receipt?

@samueljohn
Contributor

Sorry for my late response here.

If you come here by and read this issue about scipy tests failing: In short, use the --use-openblas switch or be ready to use --devel or ignore the errors as they are really more or less corner cases :-) (suggest to read the upstream issues).

And thank you for your instructions @ccwang002, perhaps ATLAS is a thing we should consider to add to homebrew/science repo? But then I need to add a --with-ATLAS switch that will naturally conflict with --with-openblas. However, I am not opposed to that. Glad to hear that ATLAS compiles without issues on a Mac.
Perhaps you are willing to provide a formula over there? I'll review a pull request and pull in (as keg_only so ATLAS does not shadow other blas libs.)

SciPy 0.13 will fix the issues with Accelerate (see discussion upstream). They did a fantastic job and the community is great. I added the 0.13b1 to the formula if you brew install scipy --devel. I generally get great performance with Apple's Accelerate and 0.13 seems to work with 0.13 just fine.
(I also added 1.8.0b1 of numpy if you use the --devel flag).

I will leave this issue open because the current release of scipy still is affected.

@samueljohn samueljohn added a commit that closed this issue Sep 4, 2013
@michaelwimmer @samueljohn michaelwimmer + samueljohn scipy use g77 ABI on Acclerate framework
apply upstream patches for ARPACK

The patches and fixes are not needed for the next releas 0.13, though.

Closes #38
Fixed #12
6812e58
@samueljohn samueljohn closed this in 6812e58 Sep 4, 2013
@samueljohn
Contributor

Thanks to @michaelwimmer for making the PR that patches numpy/scipy with the upstream fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment