TST test_corr_nearest_factor fails on Ubuntu #1850

Closed
josef-pkt opened this Issue Jul 29, 2014 · 13 comments

Projects

None yet

2 participants

@josef-pkt
Member

a test failure that has been there for some time, on i386 only (not failing on amd64)

https://launchpadlibrarian.net/180947074/buildlog_ubuntu-utopic-i386.statsmodels_0.6.0~ppa18~revno-1681~ubuntu14.10.1_UPLOADING.txt.gz

======================================================================
FAIL: statsmodels.stats.tests.test_corrpsd.Test_Factor.test_corr_nearest_factor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/buildd/statsmodels-0.6.0~ppa18~revno/debian/python-statsmodels/usr/lib/python2.7/dist-packages/statsmodels/stats/tests/test_corrpsd.py", line 239, in test_corr_nearest_factor
    assert(np.abs(mat - mat1).max() < 1e-3)
AssertionError
@josef-pkt
Member

problem: using plain assert instead of assert_allclose doesn't show the values

@kshedden any idea?

I will look into it within a few days.

@josef-pkt josef-pkt added the prio-high label Jul 29, 2014
@josef-pkt josef-pkt added this to the 0.6 milestone Jul 29, 2014
@kshedden
Contributor

I PR'd a fix of the asserts. I don't get testing errors locally. I will
keep an eye on Travis to see what happens.

On Tue, Jul 29, 2014 at 2:27 PM, Josef Perktold notifications@github.com
wrote:

problem: using plain assert instead of assert_allclose doesn't show the
values

@kshedden https://github.com/kshedden any idea?

I will look into it within a few days.


Reply to this email directly or view it on GitHub
#1850 (comment)
.

@josef-pkt
Member

PR for changing asserts is #1851

@josef-pkt
Member

new test failure message

======================================================================
FAIL: statsmodels.stats.tests.test_corrpsd.Test_Factor.test_corr_nearest_factor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/buildd/statsmodels-0.6.0~ppa18~revno/debian/python-statsmodels/usr/lib/python2.7/dist-packages/statsmodels/stats/tests/test_corrpsd.py", line 239, in test_corr_nearest_factor
    assert_allclose(mat, mat1, rtol=0.25, atol=1e-3)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 1183, in assert_allclose
    verbose=verbose, header=header)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.25, atol=0.001

(mismatch 100.0%)
 x: array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
       [  0.00000000e+00,   1.00000000e+00,   1.95129905e-02, ...,...
 y: array([[  1.00000000e+00,   8.42886695e-04,   3.89911703e-03, ...,
         -3.89911703e-03,  -8.42886695e-04,   4.66953896e-03],
       [  8.42886695e-04,   1.00000000e+00,   3.33553677e-03, ...,...

----------------------------------------------------------------------
@josef-pkt
Member

my results for dm=2
around 5e-7 close to zero, instead of failing e-2 or e-3 on python-xy ubuntu

(Pdb) x[:5,:3]
array([[ 1.        ,  0.        ,  0.        ],
       [ 0.        ,  1.        ,  0.01951299],
       [ 0.        ,  0.01951299,  1.        ],
       [ 0.        ,  0.02893583,  0.05747703],
       [ 0.        ,  0.03796404,  0.07541138]])
(Pdb) y[:5,:3]
array([[  1.00000000e+00,   3.15777121e-07,   6.30334866e-07],
       [  3.15777121e-07,   1.00000000e+00,   1.95110383e-02],
       [  6.30334866e-07,   1.95110383e-02,   1.00000000e+00],
       [  9.42258139e-07,   2.89322832e-02,   5.74728416e-02],
       [  1.25043341e-06,   3.79596895e-02,   7.54065037e-02]])
(Pdb) y[0,-3:]
array([ -6.30334866e-07,  -3.15777121e-07,  -5.05844558e-11])
@kshedden
Contributor

Attempting to resolve #1850

I don't have a machine where I can run pythonxy, and the error does not
appear on the two machines that I use (linux x86 servers).

I added asserts to check the convergence status of _spg_optim (the
underlying optimizer for cor_nearest_factor). I also attached the history
of objective function values to the object returned by cor_nearest_factor.
I could also attach the parameter value history, but that adds a lot of
weight to the return object.

If someone has the time to do this, we can compare the convergence status
and objective function history on pythonxy to other environments. Perhaps
this will clarify what is going on.

On Wed, Jul 30, 2014 at 5:53 PM, Josef Perktold notifications@github.com
wrote:

my results for dm=2
around 5e-7 close to zero, instead of failing e-2 or e-3 on python-xy
ubuntu

(Pdb) x[:5,:3]
array([[ 1. , 0. , 0. ],
[ 0. , 1. , 0.01951299],
[ 0. , 0.01951299, 1. ],
[ 0. , 0.02893583, 0.05747703],
[ 0. , 0.03796404, 0.07541138]])
(Pdb) y[:5,:3]
array([[ 1.00000000e+00, 3.15777121e-07, 6.30334866e-07],
[ 3.15777121e-07, 1.00000000e+00, 1.95110383e-02],
[ 6.30334866e-07, 1.95110383e-02, 1.00000000e+00],
[ 9.42258139e-07, 2.89322832e-02, 5.74728416e-02],
[ 1.25043341e-06, 3.79596895e-02, 7.54065037e-02]])
(Pdb) y[0,-3:]
array([ -6.30334866e-07, -3.15777121e-07, -5.05844558e-11])


Reply to this email directly or view it on GitHub
#1850 (comment)
.

@josef-pkt
Member

still fails for the two factor test case

sparse.svds for 1 factor doesn't fail
initial, first 5, objective_values in both cases don't fail

starting values seem to be ok, somewhere convergence must get stuck or enter a circle.

======================================================================
FAIL: statsmodels.stats.tests.test_corrpsd.Test_Factor.test_corr_nearest_factor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/buildd/statsmodels-0.6.0~ppa18~revno/debian/python-statsmodels/usr/lib/python2.7/dist-packages/statsmodels/stats/tests/test_corrpsd.py", line 307, in test_corr_nearest_factor
    assert_equal(rslt.Converged, True, err_msg=err_msg)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 317, in assert_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal: rank=2, niter=10001
 ACTUAL: False
 DESIRED: True
@josef-pkt
Member

It's cycling with period 7:

>>> rslt.objective_values[-20:]
[121.47755550070228, 99.009792263890006, 40.342072751551875, 22.268450441361168, 64.531986403532542, 33.022241168002274, 84.077780698529992, 121.47755550070268, 99.009792263890375, 40.342072751551996, 22.268450441361434, 64.531986403532997, 33.022241168002417, 84.07778069853012, 121.47755550070241, 99.009792263890276, 40.342072751552017, 22.268450441361459, 64.531986403533182, 33.022241168002502]
>>> np.diff(np.asarray(rslt.objective_values)[-5*7:].reshape(-1,7), axis=0)
array([[ -5.25801624e-13,   1.56319402e-13,  -1.13686838e-13,
         -4.26325641e-13,   2.84217094e-13,   4.40536496e-13,
          1.20792265e-13],
       [ -1.13686838e-13,  -1.09423581e-12,  -6.96331881e-13,
         -1.13686838e-13,  -3.19744231e-14,  -2.84217094e-14,
         -7.81597009e-14],
       [  7.10542736e-14,   3.97903932e-13,   3.69482223e-13,
          1.20792265e-13,   2.66453526e-13,   4.54747351e-13,
          1.42108547e-13],
       [  1.27897692e-13,  -2.70006240e-13,  -9.94759830e-14,
          2.13162821e-14,   2.48689958e-14,   1.84741111e-13,
          8.52651283e-14]])

plotting the objective_values it starts to fluctuate in a regular way without converging.

I got this after running it several times in the interpreter. It never failed for me running the unit tests, and it doesn't fail every time.

@kshedden Can you check if there is a condition that would prevent cycling without convergence?

@josef-pkt
Member

example: in a few runs it's failing

>>> execfile('try_corr_factor_arpack.py')
niter=10001 False
>>> execfile('try_corr_factor_arpack.py')
niter=1093 True
>>> execfile('try_corr_factor_arpack.py')
niter=1177 True
>>> execfile('try_corr_factor_arpack.py')
niter=1080 True
>>> execfile('try_corr_factor_arpack.py')
niter=1063 True
>>> execfile('try_corr_factor_arpack.py')
niter=1141 True
>>> execfile('try_corr_factor_arpack.py')
niter=1224 True
>>> execfile('try_corr_factor_arpack.py')
niter=1191 True
>>> execfile('try_corr_factor_arpack.py')
niter=10001 False
>>> execfile('try_corr_factor_arpack.py')
niter=1300 True
>>> execfile('try_corr_factor_arpack.py')
niter=1165 True
>>> execfile('try_corr_factor_arpack.py')
niter=1329 True
>>> execfile('try_corr_factor_arpack.py')
niter=1158 True
>>> execfile('try_corr_factor_arpack.py')
niter=1371 True
>>> execfile('try_corr_factor_arpack.py')
niter=1087 True
>>> execfile('try_corr_factor_arpack.py')
niter=1132 True
>>> execfile('try_corr_factor_arpack.py')
niter=10001 False
>>> 
@josef-pkt
Member

arpack sparse.svds is off the hook.
I always get the same starting values whether it's a converging or non-converging case.

@josef-pkt
Member

@kshedden
I think we can just adjust the test to add some random noise.
My guess is now that this is a edge case where that might cycle because of the deterministic structure and relationship between the factors.

If I add some random noise to the data

for j in range(dm):
        X[:,j] = np.sin(x*(j+1)) + 0.01 * np.random.randn(d)

then I don't get a convergence failure or cycling in few hundred runs of corr_nearest_factor, and the assert passes at

assert_allclose(mat, mat1, rtol=0.25, atol=1e-4, err_msg=msg)
atol is a bit tighter than currently in master

@josef-pkt
Member

another possible complication:

for the checks in the previous comment I only used the two factor version.
While running the tests for #1888 I got a hanging process that required hard kill. So it must be in one of the C or Fortran parts of numpy or scipy.
I verified that X (the data matrix) is all isfinite, so it cannot be an inf in some linear algebra.

@josef-pkt
Member

PR #1888 adding noise to the data removed the test failure on Ubuntu pythonxy testing.

I'm closing this but will open new issue with possible fragility of this function.

@josef-pkt josef-pkt closed this Aug 17, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment