ProbitCG failures #1648

Closed
jseabold opened this Issue Apr 29, 2014 · 11 comments

Projects

None yet

2 participants

@jseabold
Member

Scipy version

[~/]
[2]: version.full_version
[2]: '0.15.0.dev-54796c7'

Report

|2 $ nosetests test_discrete.py 
S........................................................................................................................S...........................................................................................................................................................................................................FF........F.FFFFF.....................................................................................................
======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_bse
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 85, in test_bse
    assert_almost_equal(self.res1.bse, self.res2.bse, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 50.0%)
x: array([ 0.6937,  0.0839,  0.5951,  2.5419])
y: array([ 0.6939,  0.0839,  0.595 ,  2.5425])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_conf_int
----------------------------------------------------------------------
Traceback (most recent call last):                                              
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest                                                                    
    self.test(*self.arg)                                                        
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 56, in test_conf_int                                           
    assert_allclose(self.res1.conf_int(), self.res2.conf_int, rtol=8e-5)        
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 1289, in assert_allclose                                                           
    verbose=verbose, header=header)                                             
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare                                                       
    raise AssertionError(msg)                                                   
AssertionError:                                                                 
Not equal to tolerance rtol=8e-05, atol=0                                       

(mismatch 100.0%)
x: array([[  0.264034,   2.983139],
    [ -0.112901,   0.215831],
    [  0.261757,   2.594653],
    [-12.423953,  -2.460038]])
y: array([[  0.265825,   2.985795],
    [ -0.112693,   0.216151],
    [  0.26008 ,   2.592585],
    [-12.43547 ,  -2.469166]])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_params
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 53, in test_params
    assert_almost_equal(self.res1.params, self.res2.params, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 100.0%)
x: array([ 1.6236,  0.0515,  1.4282, -7.442 ])
y: array([ 1.6258,  0.0517,  1.4263, -7.4523])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_predict
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 99, in test_predict
    self.res2.phat, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 68.75%)
x: array([ 0.0181,  0.0529,  0.189 ,  0.0186,  0.5529,  0.0272,  0.0185,
        0.0444,  0.1082,  0.6609,  0.0161,  0.1927,  0.322 ,  0.1942,
        0.3547,  0.0219,  0.0455,  0.0308,  0.5931,  0.6567,  0.0622,...
y: array([ 0.0182,  0.0531,  0.1899,  0.0186,  0.5546,  0.0272,  0.0185,
        0.0446,  0.1088,  0.6631,  0.0161,  0.1936,  0.3233,  0.1952,
        0.3563,  0.022 ,  0.0457,  0.0309,  0.5934,  0.6572,  0.0619,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_predict_xb
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 104, in test_predict_xb
    self.res2.yhat, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 96.875%)
x: array([-2.094 , -1.6176, -0.8815, -2.0835,  0.1331, -1.9236, -2.086 ,
    -1.7015, -1.2359,  0.415 , -2.1427, -0.868 , -0.4621, -0.8625,
    -0.3726, -2.0155, -1.6905, -1.8694,  0.2355,  0.4034, -1.537 ,...
y: array([-2.0931, -1.6157, -0.8782, -2.0842,  0.1372, -1.9231, -2.0857,
    -1.6999, -1.2329,  0.421 , -2.1419, -0.8649, -0.4584, -0.859 ,
    -0.3683, -2.0148, -1.6881, -1.8684,  0.2363,  0.4048, -1.5388,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_resid_dev
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 124, in test_resid_dev
    DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 87.5%)
x: array([-0.1913, -0.3296, -0.6473, -0.1938,  1.0886, -0.2348, -0.1932,
    -0.3015, -0.4787,  0.9101, -0.18  , -0.6543, -0.8816,  1.8104,
    -0.936 , -0.2106, -0.3051, -0.2501, -1.341 ,  0.9172, -0.3582,...
y: array([-0.1915, -0.3303, -0.649 , -0.1936,  1.0859, -0.235 , -0.1933,
    -0.302 , -0.48  ,  0.9064, -0.1802, -0.6559, -0.8838,  1.8077,
    -0.9387, -0.2108, -0.3058, -0.2503, -1.3416,  0.9163, -0.3576,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_resid_generalized
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 128, in test_resid_generalized
    self.res2.resid_generalized, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 71.875%)
x: array([-0.0454, -0.1138, -0.3336, -0.0464,  0.7151, -0.0645, -0.0461,
    -0.0982, -0.2084,  0.5538, -0.0408, -0.3391, -0.5288,  1.4161,
    -0.5768, -0.0535, -0.1001, -0.0717, -0.9536,  0.5601, -0.1306,...
y: array([-0.0455, -0.1142, -0.3349, -0.0463,  0.7126, -0.0645, -0.0462,
    -0.0984, -0.2093,  0.5506, -0.0409, -0.3403, -0.5308,  1.4134,
    -0.5792, -0.0536, -0.1006, -0.0719, -0.9542,  0.5593, -0.1302,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_zstat
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 59, in test_zstat
    assert_almost_equal(self.res1.tvalues, self.res2.z, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 100.0%)
x: array([ 2.3406,  0.6137,  2.3998, -2.9278])
y: array([ 2.3431,  0.6166,  2.397 , -2.9311])

----------------------------------------------------------------------
Ran 443 tests in 19.558s
@jseabold jseabold added this to the 0.6 milestone Apr 29, 2014
@jseabold
Member

I actually can't make CG converge for this problem without setting gtol ridiculously low. I don't understand why. There seems to be some problem, and it's not respecting gtol or maxiter. The norm of the gradient at the final iteration is certainly not less than gtol. Looks like it could be a scaling issue?

@josef-pkt
Member

duplicate (kind of)
I just reopened #109 because it fails on Windows 64 (Matthews wheels)
try maxiter=1000

@jseabold
Member

I tried maxiter=1e6. It never does more than 160 or so function calls. It
seems to think there's an ill-conditioning problem and just craps out. The
function value is very near the optimum here though. I wonder if we just
remove this test. We know, in principal that it works. I could be this
changeset for _line_search_wolfe12, but I haven't done a bisect.

@josef-pkt
Member

Ok, that's the same problem I have on Win32, but the unit tests pass because we are close enough. I didn't look at what causes it.

It's the only unit test we have for fmin_cg (AFAIK), and I'd rather figure out if we can fix the test instead of just removing it.

If it's the lineseach again, then we really need to check, because we also have the problem with the line search in bfgs in scipy > 0.12.

@jseabold
Member

Ok will leave open. It's nothing I can get deep into right now.

@josef-pkt
Member

this is an example for #1649

I have the test case as a script that I used to check different scipy binaries for Windows.

Is the failure that you have only with scipy dev version, or have you seen it before?
TestProbitCG was quite for some time, and doesn't fail on TravisCI, Debian and Ubuntu testing.

@jseabold
Member

I don't recall seeing it before today.

@jseabold
Member

I have my ordqz branch from scipy on my path, but I recently rebased it,
so something must have changed in scipy.

@jseabold
Member

Though I guess I also updated my OpenBLAS recently too, so I need to
bisect.

@jseabold
Member

I don't see it on my desktop with scipy-0.14.0.dev-1b7c11c and ATLAS.

@josef-pkt
Member

Maybe then there is still a chance it goes away.

In either case, win32 also only gets close enough but stops before reaching an optimum.
It still needs investigation, but not "right now"

@josef-pkt josef-pkt closed this in ca701e7 May 27, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment