test_arima.test_small_data: SVD fails to converge (Python 3.4 / Ubuntu 12.04) #1689

Closed
andrewclegg opened this Issue May 24, 2014 · 4 comments

Projects

None yet

2 participants

@andrewclegg
Contributor

I am attempting to install statsmodels master on Python 3.4 for the Snake Charmer project. Several tests are failing or throwing errors.

Python 3.4
Ubuntu 12.04 64 bit (virtual)
NumPy 1.8.1
SciPy 0.13.1

statsmodels revision 3adaa1a
Snake Charmer revision snake-charmer-devs/snake-charmer@72734de

To reproduce: install Snake Charmer, spin up a VM as directed in the README, log in using vagrant ssh, test as normal.

======================================================================
ERROR: statsmodels.tsa.tests.test_arima.test_small_data
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/tests/test_arima.py", line 1886, in test_small_data
    res = mod.fit(disp=0, start_params=[.1, .1, .1, .1])
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 886, in fit
    callback=callback, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py", line 390, in fit
    full_output=full_output)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/optimizer.py", line 184, in _fit
    hess=hessian)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/optimizer.py", line 378, in _fit_lbfgs
    **extra_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 185, in fmin_l_bfgs_b
    **opts)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 314, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "/usr/local/lib/python3.4/dist-packages/scipy/optimize/lbfgsb.py", line 258, in func_and_grad
    f = fun(x, *args)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py", line 369, in <lambda>
    f = lambda params, *args: -self.loglike(params, *args) / nobs
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 696, in loglike
    return self.loglike_kalman(params, set_sigma2)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/arima_model.py", line 706, in loglike_kalman
    return KalmanFilter.loglike(params, self, set_sigma2)
  File "/usr/local/lib/python3.4/dist-packages/statsmodels/tsa/kalmanf/kalmanfilter.py", line 649, in loglike
    R_mat, T_mat)
  File "kalman_loglike.pyx", line 342, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_loglike_double (statsmodels/tsa/kalmanf/kalman_loglike.c:5046)
  File "kalman_loglike.pyx", line 74, in statsmodels.tsa.kalmanf.kalman_loglike.kalman_filter_double (statsmodels/tsa/kalmanf/kalman_loglike.c:2525)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 1585, in pinv
    u, s, vt = svd(a, 0)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 1327, in svd
    u, s, vt = gufunc(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.4/dist-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge
@andrewclegg andrewclegg referenced this issue in snake-charmer-devs/snake-charmer May 24, 2014
Closed

One problem in statsmodels #14

@josef-pkt
Member

(I thought I added a comment but I don't find it anymore.)

This is another case where we are just add the edge of what should work numerically (even if it doesn't make much sense).
We had some problems with this some time ago, but it also has been quite for a while, no test failures or errors on Ubuntu, Debian or Windows.

Since I cannot replicate this, can you try to change one of the start_params and see if it helps
on line
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/test_arima.py#L1886
from
res = mod.fit(disp=0, start_params=[.1, .1, .1, .1])
to this
res = mod.fit(disp=0, start_params=[np.mean(y), .1, .1, .1])

adding np.mean(y) as starting value for the constant.

one common source for SVD did not converge are nans in the array, but I don't know if it would help to find them since this is not a regular use case.

@andrewclegg
Contributor

np.mean(y) fixed it.

@josef-pkt
Member

Good, that's an easy change, and doesn't make the test meaningless.

You could make a PR, or I'll fix it later today.

@andrewclegg andrewclegg added a commit to andrewclegg/statsmodels that referenced this issue May 27, 2014
@andrewclegg andrewclegg Fix for statsmodels/statsmodels#1689 cacc4e0
@andrewclegg
Contributor

There you go! Thanks for the tip, glad it was a quick change.

@josef-pkt josef-pkt closed this in 0b689a2 May 27, 2014
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@andrewclegg andrewclegg Fix for statsmodels/statsmodels#1689 07da936
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment