test failure in LLE

[mblondel]

I'm (still?) getting a test failure in the LLE module.

======================================================================
FAIL: test_locally_linear.test_lle_simple_grid
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest
    self.test(*self.arg)
  File "/home/mathieu/Desktop/projects/scikit-learn/sklearn/manifold/tests/test_locally_linear.py", line 67, in te
st_lle_simple_grid
    assert_lower(np.linalg.norm(X_reembedded - clf.embedding_), tol)
  File "/home/mathieu/Desktop/projects/scikit-learn/sklearn/manifold/tests/test_locally_linear.py", line 15, in as
sert_lower
    assert a < b, message
AssertionError: 0.67260012006758951 is not lower than 0.10000000000000001
>>  assert 0.67260012006758951 < 0.10000000000000001, '0.67260012006758951 is not lower than 0.10000000000000001'

The strange thing is that the test passes when test_locally_linear.py is run individually but fails when run from make test-code.

BTW, make test is supposed to run the test-code and test-doc targets but I noticed that when there is a failure in test-code, test-doc is not executed. This means that a failure in a unrelated module to your changes can hide doctest failures.

If the test is difficult to fix, we could just skip it, maybe?

[ogrisel]

Is there some use of a np.random global random state rather than using an explicitly seeded rng?

[mblondel]

@ogrisel: it seems that the tests don't use a random state. I'd rather skip the test if it's fragile. What do you think, @jakevdp?

EDIT: I meant to write "the tests don't use a global random state".

[ogrisel]

The test is using explicit seeding. Maybe the randomness comes from the arpack solver? Or it can be a consequence on the other memory bug in the ball tree implementation.

[jakevdp]

There is some randomness in ARPACK, I believe. When calling the fortran routines, one can specify a starting vector - I think that's selected randomly if not supplied.

[ogrisel]

I think we should supply a random init vector based on the random_state of the wrapping scikit-learn class to make that behavior controllable.

[mblondel]

+1 with @ogrisel's solution

[jakevdp]

+1 The starting vector can be specified with the v0 keyword to eigsh.

[amueller]

Should this be closed?

[jakevdp]

I think this test can still fail (not sure why). The random_state PR was just a way to get more control in the testing. @mblondel, do you still see this failure?

[mblondel]

Yes, I still see the failure...

[amueller]

This issue only shows with scipy 0.9 as far as I can tell.

[amueller]

Should be fixed in da022b8. Can you confirm please?

[andrewmilkowski]

Reporting it here (getting similar error during while running nosetests sklearn ) using latest git master (with the patch, verified the change in the source code, but otherwise ready to run examples and learn! thanks for the project

Environment:

[amilkowski@localhost scikit-learn]$ uname -a
Linux localhost.localdomain 3.4.3-1.fc17.x86_64 #1 SMP Mon Jun 18 19:53:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

FAIL: test_locally_linear.test_lle_simple_grid

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/opt/git/python/scikit-learn/sklearn/manifold/tests/test_locally_linear.py", line 64, in test_lle_simple_grid
assert_less(np.linalg.norm(X_reembedded - clf.embedding_), tol)
AssertionError: 0.51937584624682986 not less than 0.1
"""Fail immediately, with the given message."""

raise self.failureException('0.51937584624682986 not less than 0.1')

also dumping scipy version if it helps

[amilkowski@localhost scikit-learn]$ sudo yum list scipy

Installed Packages
scipy.x86_64 0.10.1-1.fc17 @fedora

[amueller]

This is hopefully fixed in #1039

[amueller]

Fixed in #1039. Thanks @jakevdp

[mblondel]

@amueller You have a good memory :) I was searching for this ticket to close it.