Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_arima.test_small_data: SVD fails to converge (Python 3.4 / Ubuntu 12.04) #1689

Closed
andrewclegg opened this issue May 24, 2014 · 4 comments

Comments

Projects
None yet
2 participants
@andrewclegg
Copy link
Contributor

commented May 24, 2014

I am attempting to install statsmodels master on Python 3.4 for the Snake Charmer project. Several tests are failing or throwing errors.

Python 3.4
Ubuntu 12.04 64 bit (virtual)
NumPy 1.8.1
SciPy 0.13.1

statsmodels revision 3adaa1a
Snake Charmer revision snake-charmer-devs/snake-charmer@72734de

To reproduce: install Snake Charmer, spin up a VM as directed in the README, log in using vagrant ssh, test as normal.

======================================================================
ERROR: statsmodels.tsa.tests.test_arima.test_small_data
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/tests/test_arima.py", line 1886, in test_small_data
    res = mod.fit(disp=0, start_params=[.1, .1, .1, .1])
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 886, in fit
    callback=callback, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py", line 390, in fit
    full_output=full_output)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/optimizer.py", line 184, in _fit
    hess=hessian)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/optimizer.py", line 378, in _fit_lbfgs
    **extra_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 185, in fmin_l_bfgs_b
    **opts)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 314, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 258, in func_and_grad
    f = fun(x, *args)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py", line 369, in <lambda>
    f = lambda params, *args: -self.loglike(params, *args) / nobs
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 696, in loglike
    return self.loglike_kalman(params, set_sigma2)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 706, in loglike_kalman
    return KalmanFilter.loglike(params, self, set_sigma2)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/kalmanf/kalmanfilter.py", line 649, in loglike
    R_mat, T_mat)
  File "kalman_loglike.pyx", line 342, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_loglike_double (statsmodels/tsa/kalmanf/kalman_loglike.c:5046)
  File "kalman_loglike.pyx", line 74, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_filter_double (statsmodels/tsa/kalmanf/kalman_loglike.c:2525)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 1585, in pinv
    u, s, vt = svd(a, 0)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 1327, in svd
    u, s, vt = gufunc(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge
@josef-pkt

This comment has been minimized.

Copy link
Member

commented May 25, 2014

(I thought I added a comment but I don't find it anymore.)

This is another case where we are just add the edge of what should work numerically (even if it doesn't make much sense).
We had some problems with this some time ago, but it also has been quite for a while, no test failures or errors on Ubuntu, Debian or Windows.

Since I cannot replicate this, can you try to change one of the start_params and see if it helps
on line
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/test_arima.py#L1886
from
res = mod.fit(disp=0, start_params=[.1, .1, .1, .1])
to this
res = mod.fit(disp=0, start_params=[np.mean(y), .1, .1, .1])

adding np.mean(y) as starting value for the constant.

one common source for SVD did not converge are nans in the array, but I don't know if it would help to find them since this is not a regular use case.

@andrewclegg

This comment has been minimized.

Copy link
Contributor Author

commented May 27, 2014

np.mean(y) fixed it.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented May 27, 2014

Good, that's an easy change, and doesn't make the test meaningless.

You could make a PR, or I'll fix it later today.

andrewclegg added a commit to andrewclegg/statsmodels that referenced this issue May 27, 2014

@andrewclegg

This comment has been minimized.

Copy link
Contributor Author

commented May 27, 2014

There you go! Thanks for the tip, glad it was a quick change.

@josef-pkt josef-pkt closed this in 0b689a2 May 27, 2014

PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014

PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.