Test Failure in LLE on ubuntu 12.10 #1632

Closed
amueller opened this Issue Jan 29, 2013 · 13 comments

Comments

Projects
None yet
5 participants
Owner

amueller commented Jan 29, 2013

So my box in uni got upgraded and now I have this test failure:

======================================================================
FAIL: sklearn.manifold.tests.test_locally_linear.test_lle_simple_grid
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/local/lamueller/checkout/scikit-learn/sklearn/manifold/tests/test_locally_linear.py", line 68, in test_lle_simple_grid
    assert_less(np.linalg.norm(X_reembedded - clf.embedding_), tol)
AssertionError: 0.12820616722921346 not less than 0.1
    """Fail immediately, with the given message."""
>>  raise self.failureException('0.12820616722921346 not less than 0.1')

scipy 0.10.1, 32bit.

Owner

agramfort commented Jan 29, 2013

FYI I have the same failing test on my 32bits install on my mac with scipy
0.10. You're not alone :)

Member

jakevdp commented Jan 30, 2013

My guess is this is due to ARPACK - it's been notoriously unstable between systems; so much so that scipy devs have disabled the 32-bit wrappers starting in 0.10.1 (see http://docs.scipy.org/doc/scipy-0.10.1/reference/release.0.10.1.html). If you're using an earlier scipy version on a 32-bit system, then that's probably the issue. Even in newer versions, I wouldn't be surprised if ARPACK is causing problems.

Owner

amueller commented Feb 11, 2013

So what is the fix? Less strict test? Less strict test if we have 32bit scipy (can we test that, maybe via numpy.distutils.system_info.platform_bits)?

Member

jakevdp commented Feb 12, 2013

Hard to say... ARPACK is so unpredictable. I suppose a less-strict test could work here, but there still might be errors on other systems out there.

Owner

amueller commented Feb 12, 2013

We do really strive for no errors, so I guess we should fix all that we know about. (for some value of "fix")

Owner

amueller commented Feb 13, 2013

Can't reproduce any more. @agramfort do you still see it? Not sure what changed in my box. If you do I guess I'll change the test a bit. Do you have the same numbers?

Owner

agramfort commented Feb 16, 2013

yes i can :

FAIL: test_locally_linear.test_lle_simple_grid

Traceback (most recent call last):
File
"/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/nose/case.py",
line 187, in runTest
self.test(*self.arg)
File
"/Users/alex/work/src/scikit-learn/sklearn/manifold/tests/test_locally_linear.py",
line 63, in test_lle_simple_grid
reconstruction_error, decimal=1)
File
"/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/numpy/testing/utils.py",
line 468, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 1 decimals
ACTUAL: 1.9482019423123158e-05
DESIRED: 0.096091666204536866

Owner

amueller commented Feb 16, 2013

that is pretty far of... wait, your error is way smaller than expected. Is that right?

Owner

agramfort commented Feb 16, 2013

the broken line is:

    assert_almost_equal(clf.reconstruction_error_,
                        reconstruction_error, decimal=1)

I ran nose on a vanilla EC2 64 bit Ubuntu 12.10 and received no errors. The full install and test log are available here:
https://gist.github.com/devinshields/5523616

Owner

amueller commented May 6, 2013

@davinshields the problem is 32bit related

Sorry, here are the 32 bit results - the tests are still failing but in a different spot. A full dump is here:
https://gist.github.com/devinshields/5523760

FAIL: sklearn.tests.test_common.test_transformers

Traceback (most recent call last):
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/sklearn/tests/test_common.py", line 230, in test_transformers
"fit_transform not correct in %s" % Trans)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 812, in assert_array_almost_equal
header=('Arrays are not almost equal to %d decimals' % decimal))
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 645, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 2 decimals
fit_transform not correct in <class 'sklearn.pls.CCA'>
(mismatch 50.0%)
x: array([[ 0.97459066, -0.37275475],
[-0.97199043, 0.22191067],
[-0.97199043, 0.22191067],...
y: array([[ 9.74590662e-01, -5.79470839e-17],
[ -9.71990427e-01, -5.85363968e-17],
[ -9.71990427e-01, -5.85363968e-17],...


Ran 1598 tests in 107.531s

FAILED (SKIP=15, failures=1)

@amueller amueller added this to the 0.15.1 milestone Jul 18, 2014

Owner

kastnerkyle commented Jul 21, 2014

Closing as duplicate - trying to organize all of the 32bit numerical stability bugs in one place, so we can squash them all :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment