GMM poisson test failures #1420

Closed
josef-pkt opened this Issue Feb 22, 2014 · 10 comments

Projects

None yet

3 participants

@josef-pkt
Member

I upgraded to scipy 0.13.3,

two test failures with gmm_poisson, that I have never seen before
they look like optimization problems

which optimizer does this use?

TODO: check how we signal convergence failures

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddOnestep.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 94, in test_basic
    assert_allclose(res1.params, res2.params, rtol=rtol, atol=0)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=5e-06, atol=0

(mismatch 100.0%)
 x: array([ 1.09152486,  0.48803911,  1.38824487, -0.31788507,  0.17899104,
        1.23690265,  1.84531486, -0.63985573])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddOnestep.test_other
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 105, in test_other
    assert_allclose(res1.q, res2.Q, rtol=atol, atol=rtol)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-09, atol=5e-06

(mismatch 100.0%)
 x: array(302509659335.9898)
 y: array(4.59536484786e-20)

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddTwostep.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 94, in test_basic
    assert_allclose(res1.params, res2.params, rtol=rtol, atol=0)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=5e-06, atol=0

(mismatch 100.0%)
 x: array([ 1.04006628,  0.4680506 ,  1.23664852, -0.69841642, -0.78233345,
        1.17508794,  1.45528787, -0.72049278])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])
@josef-pkt josef-pkt added the type-bug label Feb 22, 2014
@josef-pkt
Member

tough optimization problem to solve

running fmin i.e. optim_method='nm' first gets us in the right neighborhood for fmin_bfgs to converge also. see for example line 28 in http://nbviewer.ipython.org/gist/josef-pkt/6890383 for similar problem.

The only change in scipy that might affect fmin_bfgs are the changes to the line searches
scipy/scipy#507

Let's see if the test failure persists, then add optim_method='nm' as pre-estimator

res0 = mod.fit(start, maxiter=0, inv_weights=w0inv,
                optim_method='nm',
                wargs={'centered':False})

start = res0.params
res1 = mod.fit(start, maxiter=0, inv_weights=w0inv,
                optim_method='bfgs', optim_args={'gtol':1e-8},
                wargs={'centered':False})
print res0.params
@jseabold
Member

Ah, good. I've been seeing these locally but wasn't sure if it was my problem.

@josef-pkt
Member

Ok, If you also see them, then I will fix the unit tests soon. They are a "feature" of scipy 0.13.?

In general, I was thinking again about whether we should add support for switching optimizers.
GMM does not yet use our optimization wrapper (no loglike).

@jseabold
Member

I would like to add this to #1121. I'm hopefully about to put to bed two big projects which might leave me with a few sprint days here and there to finish up my WIPs.

@jseabold
Member

Maybe worth asking pv why this now fails? Could be a corner case?

@argriffing

I'm not sure if this would help, but I've empirically observed that lbfgsb is 'better' than bfgs even for unbounded problems. I've not systematically tested this.

fwiw I can reproduce your test failure

(mismatch 100.0%)
 x: array([ 1.04006628,  0.4680506 ,  1.23664852, -0.69841642, -0.78233345,
        1.17508794,  1.45528787, -0.72049278])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])

with the scipy development branch

@jseabold
Member

I agree. That's why I used it for the default in ARMA.

@argriffing

I tried to isolate the optimization out of statsmodels but it was too deep to go safely without spelunking gear

@josef-pkt
Member

I had started an email to scipy-dev but stopped. The only change that I could find is for the stepsize.
The problem is a bit nasty, gradient and hessian are very large at the current starting values.
I didn't try to figure out where bfgs runs into problems, my first guess that stepsize is too large, is probably wrong.

Given that I'm on scipy 0.13. now, I'd like to see if the new newton methods are good.

@argriffing

Given that I'm on scipy 0.13. now, I'd like to see if the new newton methods are good.

The dogleg uses the full hessian (not just hessian-vector product) and it is better if your function is not too gnarly. The trust-ncg uses the hessian through its hessian-vector product and works quite well even for gnarly functions.

@josef-pkt josef-pkt closed this in 5f05f42 Apr 4, 2014
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@josef-pkt josef-pkt TST: gmm-poisson better starting value for scipy 0.13 closes #1420 ee1fdf6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment