Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProbitCG failures #1648

Closed
jseabold opened this issue Apr 29, 2014 · 11 comments

Comments

Projects
None yet
2 participants
@jseabold
Copy link
Member

commented Apr 29, 2014

Scipy version

[~/]
[2]: version.full_version
[2]: '0.15.0.dev-54796c7'

Report

|2 $ nosetests test_discrete.py 
S........................................................................................................................S...........................................................................................................................................................................................................FF........F.FFFFF.....................................................................................................
======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_bse
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 85, in test_bse
    assert_almost_equal(self.res1.bse, self.res2.bse, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 50.0%)
x: array([ 0.6937,  0.0839,  0.5951,  2.5419])
y: array([ 0.6939,  0.0839,  0.595 ,  2.5425])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_conf_int
----------------------------------------------------------------------
Traceback (most recent call last):                                              
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest                                                                    
    self.test(*self.arg)                                                        
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 56, in test_conf_int                                           
    assert_allclose(self.res1.conf_int(), self.res2.conf_int, rtol=8e-5)        
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 1289, in assert_allclose                                                           
    verbose=verbose, header=header)                                             
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare                                                       
    raise AssertionError(msg)                                                   
AssertionError:                                                                 
Not equal to tolerance rtol=8e-05, atol=0                                       

(mismatch 100.0%)
x: array([[  0.264034,   2.983139],
    [ -0.112901,   0.215831],
    [  0.261757,   2.594653],
    [-12.423953,  -2.460038]])
y: array([[  0.265825,   2.985795],
    [ -0.112693,   0.216151],
    [  0.26008 ,   2.592585],
    [-12.43547 ,  -2.469166]])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_params
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 53, in test_params
    assert_almost_equal(self.res1.params, self.res2.params, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 100.0%)
x: array([ 1.6236,  0.0515,  1.4282, -7.442 ])
y: array([ 1.6258,  0.0517,  1.4263, -7.4523])

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_predict
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 99, in test_predict
    self.res2.phat, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 68.75%)
x: array([ 0.0181,  0.0529,  0.189 ,  0.0186,  0.5529,  0.0272,  0.0185,
        0.0444,  0.1082,  0.6609,  0.0161,  0.1927,  0.322 ,  0.1942,
        0.3547,  0.0219,  0.0455,  0.0308,  0.5931,  0.6567,  0.0622,...
y: array([ 0.0182,  0.0531,  0.1899,  0.0186,  0.5546,  0.0272,  0.0185,
        0.0446,  0.1088,  0.6631,  0.0161,  0.1936,  0.3233,  0.1952,
        0.3563,  0.022 ,  0.0457,  0.0309,  0.5934,  0.6572,  0.0619,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_predict_xb
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 104, in test_predict_xb
    self.res2.yhat, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 96.875%)
x: array([-2.094 , -1.6176, -0.8815, -2.0835,  0.1331, -1.9236, -2.086 ,
    -1.7015, -1.2359,  0.415 , -2.1427, -0.868 , -0.4621, -0.8625,
    -0.3726, -2.0155, -1.6905, -1.8694,  0.2355,  0.4034, -1.537 ,...
y: array([-2.0931, -1.6157, -0.8782, -2.0842,  0.1372, -1.9231, -2.0857,
    -1.6999, -1.2329,  0.421 , -2.1419, -0.8649, -0.4584, -0.859 ,
    -0.3683, -2.0148, -1.6881, -1.8684,  0.2363,  0.4048, -1.5388,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_resid_dev
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 124, in test_resid_dev
    DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 87.5%)
x: array([-0.1913, -0.3296, -0.6473, -0.1938,  1.0886, -0.2348, -0.1932,
    -0.3015, -0.4787,  0.9101, -0.18  , -0.6543, -0.8816,  1.8104,
    -0.936 , -0.2106, -0.3051, -0.2501, -1.341 ,  0.9172, -0.3582,...
y: array([-0.1915, -0.3303, -0.649 , -0.1936,  1.0859, -0.235 , -0.1933,
    -0.302 , -0.48  ,  0.9064, -0.1802, -0.6559, -0.8838,  1.8077,
    -0.9387, -0.2108, -0.3058, -0.2503, -1.3416,  0.9163, -0.3576,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_resid_generalized
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 128, in test_resid_generalized
    self.res2.resid_generalized, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 71.875%)
x: array([-0.0454, -0.1138, -0.3336, -0.0464,  0.7151, -0.0645, -0.0461,
    -0.0982, -0.2084,  0.5538, -0.0408, -0.3391, -0.5288,  1.4161,
    -0.5768, -0.0535, -0.1001, -0.0717, -0.9536,  0.5601, -0.1306,...
y: array([-0.0455, -0.1142, -0.3349, -0.0463,  0.7126, -0.0645, -0.0462,
    -0.0984, -0.2093,  0.5506, -0.0409, -0.3403, -0.5308,  1.4134,
    -0.5792, -0.0536, -0.1006, -0.0719, -0.9542,  0.5593, -0.1302,...

======================================================================
FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_zstat
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/skipper/.local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
File "/home/skipper/statsmodels/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 59, in test_zstat
    assert_almost_equal(self.res1.tvalues, self.res2.z, DECIMAL_4)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 470, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 836, in assert_array_almost_equal
    precision=decimal)
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 660, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 100.0%)
x: array([ 2.3406,  0.6137,  2.3998, -2.9278])
y: array([ 2.3431,  0.6166,  2.397 , -2.9311])

----------------------------------------------------------------------
Ran 443 tests in 19.558s

@jseabold jseabold added this to the 0.6 milestone Apr 29, 2014

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

I actually can't make CG converge for this problem without setting gtol ridiculously low. I don't understand why. There seems to be some problem, and it's not respecting gtol or maxiter. The norm of the gradient at the final iteration is certainly not less than gtol. Looks like it could be a scaling issue?

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2014

duplicate (kind of)
I just reopened #109 because it fails on Windows 64 (Matthews wheels)
try maxiter=1000

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

I tried maxiter=1e6. It never does more than 160 or so function calls. It
seems to think there's an ill-conditioning problem and just craps out. The
function value is very near the optimum here though. I wonder if we just
remove this test. We know, in principal that it works. I could be this
changeset for _line_search_wolfe12, but I haven't done a bisect.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2014

Ok, that's the same problem I have on Win32, but the unit tests pass because we are close enough. I didn't look at what causes it.

It's the only unit test we have for fmin_cg (AFAIK), and I'd rather figure out if we can fix the test instead of just removing it.

If it's the lineseach again, then we really need to check, because we also have the problem with the line search in bfgs in scipy > 0.12.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

Ok will leave open. It's nothing I can get deep into right now.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2014

this is an example for #1649

I have the test case as a script that I used to check different scipy binaries for Windows.

Is the failure that you have only with scipy dev version, or have you seen it before?
TestProbitCG was quite for some time, and doesn't fail on TravisCI, Debian and Ubuntu testing.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

I don't recall seeing it before today.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

I have my ordqz branch from scipy on my path, but I recently rebased it,
so something must have changed in scipy.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

Though I guess I also updated my OpenBLAS recently too, so I need to
bisect.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2014

I don't see it on my desktop with scipy-0.14.0.dev-1b7c11c and ATLAS.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2014

Maybe then there is still a chance it goes away.

In either case, win32 also only gets close enough but stops before reaching an optimum.
It still needs investigation, but not "right now"

@josef-pkt josef-pkt closed this in ca701e7 May 27, 2014

PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014

Merge pull request statsmodels#1699 from josef-pkt/reparameterize
TST/ENH StandardizeTransform, reparameterize TestProbitCG closes statsmodels#1648 closes statsmodels#1690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.