Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GMM poisson test failures #1420

Closed
josef-pkt opened this issue Feb 22, 2014 · 10 comments

Comments

Projects
None yet
3 participants
@josef-pkt
Copy link
Member

commented Feb 22, 2014

I upgraded to scipy 0.13.3,

two test failures with gmm_poisson, that I have never seen before
they look like optimization problems

which optimizer does this use?

TODO: check how we signal convergence failures

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddOnestep.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 94, in test_basic
    assert_allclose(res1.params, res2.params, rtol=rtol, atol=0)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=5e-06, atol=0

(mismatch 100.0%)
 x: array([ 1.09152486,  0.48803911,  1.38824487, -0.31788507,  0.17899104,
        1.23690265,  1.84531486, -0.63985573])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddOnestep.test_other
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 105, in test_other
    assert_allclose(res1.q, res2.Q, rtol=atol, atol=rtol)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-09, atol=5e-06

(mismatch 100.0%)
 x: array(302509659335.9898)
 y: array(4.59536484786e-20)

======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm_poisson.TestGMMAddTwostep.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\sandbox\regression\tests\test_gmm_poisson.py", line 94, in test_basic
    assert_allclose(res1.params, res2.params, rtol=rtol, atol=0)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=5e-06, atol=0

(mismatch 100.0%)
 x: array([ 1.04006628,  0.4680506 ,  1.23664852, -0.69841642, -0.78233345,
        1.17508794,  1.45528787, -0.72049278])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])

@josef-pkt josef-pkt added the type-bug label Feb 22, 2014

@josef-pkt

This comment has been minimized.

Copy link
Member Author

commented Feb 22, 2014

tough optimization problem to solve

running fmin i.e. optim_method='nm' first gets us in the right neighborhood for fmin_bfgs to converge also. see for example line 28 in http://nbviewer.ipython.org/gist/josef-pkt/6890383 for similar problem.

The only change in scipy that might affect fmin_bfgs are the changes to the line searches
scipy/scipy#507

Let's see if the test failure persists, then add optim_method='nm' as pre-estimator

res0 = mod.fit(start, maxiter=0, inv_weights=w0inv,
                optim_method='nm',
                wargs={'centered':False})

start = res0.params
res1 = mod.fit(start, maxiter=0, inv_weights=w0inv,
                optim_method='bfgs', optim_args={'gtol':1e-8},
                wargs={'centered':False})
print res0.params
@jseabold

This comment has been minimized.

Copy link
Member

commented Feb 25, 2014

Ah, good. I've been seeing these locally but wasn't sure if it was my problem.

@josef-pkt

This comment has been minimized.

Copy link
Member Author

commented Feb 25, 2014

Ok, If you also see them, then I will fix the unit tests soon. They are a "feature" of scipy 0.13.?

In general, I was thinking again about whether we should add support for switching optimizers.
GMM does not yet use our optimization wrapper (no loglike).

@jseabold

This comment has been minimized.

Copy link
Member

commented Feb 25, 2014

I would like to add this to #1121. I'm hopefully about to put to bed two big projects which might leave me with a few sprint days here and there to finish up my WIPs.

@jseabold

This comment has been minimized.

Copy link
Member

commented Feb 25, 2014

Maybe worth asking pv why this now fails? Could be a corner case?

@argriffing

This comment has been minimized.

Copy link

commented Feb 25, 2014

I'm not sure if this would help, but I've empirically observed that lbfgsb is 'better' than bfgs even for unbounded problems. I've not systematically tested this.

fwiw I can reproduce your test failure

(mismatch 100.0%)
 x: array([ 1.04006628,  0.4680506 ,  1.23664852, -0.69841642, -0.78233345,
        1.17508794,  1.45528787, -0.72049278])
 y: array([ 0.62093806,  0.688957  ,  0.25750627, -0.05352997,  0.03106248,
        0.14175366,  0.23128095,  0.34763567])

with the scipy development branch

@jseabold

This comment has been minimized.

Copy link
Member

commented Feb 25, 2014

I agree. That's why I used it for the default in ARMA.

@argriffing

This comment has been minimized.

Copy link

commented Feb 25, 2014

I tried to isolate the optimization out of statsmodels but it was too deep to go safely without spelunking gear

@josef-pkt

This comment has been minimized.

Copy link
Member Author

commented Feb 25, 2014

I had started an email to scipy-dev but stopped. The only change that I could find is for the stepsize.
The problem is a bit nasty, gradient and hessian are very large at the current starting values.
I didn't try to figure out where bfgs runs into problems, my first guess that stepsize is too large, is probably wrong.

Given that I'm on scipy 0.13. now, I'd like to see if the new newton methods are good.

@argriffing

This comment has been minimized.

Copy link

commented Feb 25, 2014

Given that I'm on scipy 0.13. now, I'd like to see if the new newton methods are good.

The dogleg uses the full hessian (not just hessian-vector product) and it is better if your function is not too gnarly. The trust-ncg uses the hessian through its hessian-vector product and works quite well even for gnarly functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.