Newton regularize #1847

Merged
merged 4 commits into from Jul 28, 2014

Projects

None yet

3 participants

@josef-pkt
Member

our optimize._fit_newton is very plain and fails if the Hessian is not invertible

see #953 for the suggestion to use pinv

This PR instead introduces a Ridge regularization, i.e. add lambda * eye

I'm not sure how lambda should be set.
The current lambda is at least 1e-10, which I needed to avoid the failing cases in #1845. Making it only depend on median or max of the diagonal elements didn't work in that case.

Problem in emplike #1845 is that _fit_newton is hardcoded as the only optimizer used in the inner optimization.

josef-pkt added some commits Jul 27, 2014
@josef-pkt josef-pkt Bug comment: commented out code, weigths don't sum to 1 see #1845 cad804f
@josef-pkt josef-pkt REF: _fit_newton add Ridge correction to Hessian, see also #953
f9fd4e6
@josef-pkt
Member

3 failures in origin regress.
My impression is that the precision is too high. llf agrees at 6 significant digits, but not at required decimals, diff between two llf differs at second digit.
I doubt that the reference results are "more correct" than the new results.

no failures anywhere else, discrete and other emplike that use _fit_newton

@j-grana6 Do you have an idea how reliable the el_results.OriginResults are?

======================================================================
FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_ci_beta
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 41, in test_ci_beta
    assert_almost_equal(llf_low, self.res2.test_llf_conf, 4)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals
 ACTUAL: -1719.8767733239681
 DESIRED: -1719.879077
======================================================================
FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_hypothesis_beta1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 33, in test_hypothesis_beta1
    self.res2.test_llf_hypoth,4)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals
 ACTUAL: 3.5903191374052525
 DESIRED: 3.6696860000001834
======================================================================
FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_llf
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 29, in test_llf
    assert_almost_equal(self.res1.llf_el, self.res2.test_llf_hat, 4)
  File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals
 ACTUAL: -1717.956043912973
 DESIRED: -1717.95833
----------------------------------------------------------------------
Ran 3116 tests in 768.058s
FAILED (SKIP=22, failures=3)
@j-grana6
Contributor

On 07/27/2014 01:37 AM, Josef Perktold wrote:

3 failures in origin regress.
My impression is that the precision is too high. llf agrees at 6
significant digits, but not at required decimals, diff between two llf
differs at second digit.
I doubt that the reference results are "more correct" than the new
results.

no failures anywhere else, discrete and other emplike that use _fit_newton

@j-grana6 https://github.com/j-grana6 Do you have an idea how
reliable the |el_results.OriginResults| are?

hmmm.. the only one that scares me a bit is test_hypothesis_beta1. Let
me see if maybe the tolerance
is too loose.
|======================================================================

FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_ci_beta

Traceback (most recent call last):
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 41, in test_ci_beta
assert_almost_equal(llf_low, self.res2.test_llf_conf, 4)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 4 decimals
ACTUAL: -1719.8767733239681

DESIRED: -1719.879077

FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_hypothesis_beta1

Traceback (most recent call last):
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 33, in test_hypothesis_beta1
self.res2.test_llf_hypoth,4)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 4 decimals
ACTUAL: 3.5903191374052525

DESIRED: 3.6696860000001834

FAIL: statsmodels.emplike.tests.test_origin.TestOrigin.test_llf

Traceback (most recent call last):
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/emplike/tests/test_origin.py", line 29, in test_llf
assert_almost_equal(self.res1.llf_el, self.res2.test_llf_hat, 4)
File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 4 decimals
ACTUAL: -1717.956043912973

DESIRED: -1717.95833

Ran 3116 tests in 768.058s
FAILED (SKIP=22, failures=3)
|


Reply to this email directly or view it on GitHub
#1847 (comment).

@josef-pkt
Member

The point here is that everything is at the optimization tolerance, AFAICS

>>> fitted.params - [ 0.        ,  0.00351813]
array([  0.00000000e+00,   1.39605537e-06])

So small differences in convergence can have a larger impact on the llr, i.e. diff of two large values that are pretty close.
My guess is that the matlab version also will have roughly the same precision, unless you increased it when you produced the reference results.

The problem for investigating things like this is that the actual optimization is hidden behind several layers without storing any of the convergence results.

@j-grana6
Contributor

On 07/27/2014 11:09 AM, Josef Perktold wrote:

The point here is that everything is at the optimization tolerance, AFAICS

|>>> fitted.params - [ 0. , 0.00351813]
array([ 0.00000000e+00, 1.39605537e-06])
|

So small differences in convergence can have a larger impact on the
llr, i.e. diff of two large values that are pretty close.
My guess is that the matlab version also will have roughly the same
precision, unless you increased it when you produced the reference
results.

The problem for investigating things like this is that the actual
optimization is hidden behind several layers without storing any of
the convergence results.

And yet another problem is that these tests complete successfully in .08
seconds on my machine... I'll need a couple days to track this down. Do
you have any suggestions on where to start?


Reply to this email directly or view it on GitHub
#1847 (comment).

@josef-pkt
Member

What's the problem in the time that it takes to finish the tests?

The best would be to add or change the test case. Do you still have the matlab code?
If we can add another test_hypothesis_beta1 with a larger difference in the parameter, then we would get a larger llr to compare.

I don't see any bug in the code, however the layers of optimization are a bit fragile, and we would need to refactor it to have more optimization options and get more results about the optimization itself.

I'd like to merge this (today), because I think it will solve the test failures on Ubuntu and some Debian machines. After it is merged, Ubuntu and Debian testing will pick it up.
If it doesn't solve the problems, then we revert or change it again.

@j-grana6
Contributor

On 07/27/2014 10:17 PM, Josef Perktold wrote:

What's the problem in the time that it takes to finish the tests?

No problem in the time it takes, it's just that it seems to be
functioning very well on my computer and I cannot reproduce the error.

The best would be to add or change the test case. Do you still have
the matlab code?
If we can add another |test_hypothesis_beta1| with a larger difference
in the parameter, then we would get a larger |llr| to compare.

I don't see any bug in the code, however the layers of optimization
are a bit fragile, and we would need to refactor it to have more
optimization options and get more results about the optimization itself.

I'd like to merge this (today),

Sorry, I will need a bit of time before I can get to this. I have to
dig up the old matlab code as well as see exactly why it is failing on
some machines and not others.

because I think it will solve the test failures on Ubuntu and some
Debian machines. After it is merged, Ubuntu and Debian testing will
pick it up.
If it doesn't solve the problems, then we revert or change it again.


Reply to this email directly or view it on GitHub
#1847 (comment).

@josef-pkt
Member

@j-grana6

I'm changing the Ridge factor, so that the result for origin regress test remains unchanged. So we shouldn't see this failure anymore after my next push.

The diag of the Hessian in this example is hess-diag [ 2.07817294e+02 3.49076107e+11]
which means a Ridge factor that depends on the max of the diagonal doesn't work in cases like this.

josef-pkt added some commits Jul 28, 2014
@josef-pkt josef-pkt REF: _fit_newton, add ridge_factor option, default 1e-10 8c32d76
@josef-pkt josef-pkt REF _fit_newton use np.asarray(hess)
47c0f98
@josef-pkt
Member

change in last commit 47c0f98 introduces asarray(hess) since the user supplied function could return array_like (list of list in one test cases)

@josef-pkt
Member

The new keyword argument ridge_factor should maybe go into kwargs. ?

@josef-pkt
Member

copy for reference, see #1316 and #1831

result of failure on Ubuntu python-xy
https://launchpadlibrarian.net/180475707/buildlog_ubuntu-trusty-i386.statsmodels_0.6.0~ppa18~revno-1680~ubuntu14.04.1_UPLOADING.txt.gz

======================================================================
FAIL: statsmodels.emplike.tests.test_regression.TestRegressionNM.test_ci_beta2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/buildd/statsmodels-0.6.0~ppa18~revno/debian/python-statsmodels/usr/lib/python2.7/dist-packages/statsmodels/emplike/tests/test_regression.py", line 150, in test_ci_beta2
    assert_almost_equal(beta2ci, self.res2.test_ci_beta2, 6)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 454, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 811, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 6 decimals

(mismatch 50.0%)
 x: array([ 0.60120459,  2.18481462])
 y: array([ 0.60120459,  2.18470794])
@coveralls

Coverage Status

Coverage decreased (-0.0%) when pulling 47c0f98 on josef-pkt:newton_regularize into a40711c on statsmodels:master.

@josef-pkt
Member

all green, merging

if this helps depend on Debian and Ubuntu testing.

@josef-pkt josef-pkt merged commit 48f7a21 into statsmodels:master Jul 28, 2014

2 checks passed

continuous-integration/appveyor AppVeyor build succeeded
Details
continuous-integration/travis-ci The Travis CI build passed
Details
@josef-pkt josef-pkt deleted the josef-pkt:newton_regularize branch Jul 29, 2014
@josef-pkt josef-pkt added the PR label Aug 11, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment