statsmodels.tsa.stattools.coint: UnboundLocalError: local variable 'bestlag' referenced before assignment

[rjsdotorg]

Describe the bug

If bestlag=0 and autolag='t-stat' an UnboundLocalError results.
If bestlag is >0 and autolag='t-stat' results are normal.
An assert or Warning might be nice.

Code Sample, a copy-pastable example if possible

Oddly, this:

                   coint(   np.random.random(1000),   np.random.random(1000),   trend='c', method='aeg', maxlag=0, autolag='t-stat', return_results=False)

does not throw an error, but the same args with real accelerometer time series data does.

       coint(
               arr[:sdp, i, ch], 
               arr[:sdp, j, ch], 
               trend='c', method='aeg', maxlag=0, autolag='t-stat', return_results=False)

Traceback (most recent call last):
 File "E:\projects\Jan Medical\software\corr_surface\segment_2_34.py", line 961, in <module>
   main()
 File "E:\projects\Jan Medical\software\corr_surface\segment_2_34.py", line 883, in main
   OnSegment(dataObj, filename, textData, recurseLevel=0)
 File "E:\projects\Jan Medical\software\corr_surface\segment_2_34.py", line 352, in OnSegment
   coint_res = coint(
 File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 1392, in coint
   res_adf = adfuller(res_co.resid, maxlag=maxlag, autolag=autolag,
 File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 267, in adfuller
   icbest, bestlag = _autolag(OLS, xdshort, fullRHS, startlag,
 File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 115, in _autolag
   return icbest, bestlag
nboundLocalError: local variable 'bestlag' referenced before assignment

Expected Output

I'd expect a Warning or AssertionError

Output of import statsmodels.api as sm; sm.show_versions()

import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS

Python: 3.8.2.final.0

statsmodels

Installed: 0.11.1 (C:\Python38\lib\site-packages\statsmodels)

Required Dependencies

cython: Not installed
numpy: 1.19.0 (C:\Python38\lib\site-packages\numpy)
scipy: 1.4.1 (C:\Python38\lib\site-packages\scipy)
pandas: 1.0.3 (C:\Python38\lib\site-packages\pandas)
dateutil: 2.8.1 (C:\Python38\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Python38\lib\site-packages\patsy)

Optional Dependencies

matplotlib: 3.2.0 (C:\Python38\lib\site-packages\matplotlib)
backend: TkAgg
cvxopt: Not installed
joblib: 0.14.1 (C:\Python38\lib\site-packages\joblib)

Developer Tools

IPython: 7.15.0 (C:\Python38\lib\site-packages\IPython)
jinja2: 2.11.2 (C:\Python38\lib\site-packages\jinja2)
sphinx: Not installed
pygments: 2.6.1 (C:\Python38\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed

[bashtage]

I cannot reproduce on 0.12. What version are you using?

[bashtage]

Seems impossible to raise in 0.12. I'm going to close, but please post if you are experiencing this on 0.12.

[rjsdotorg]

It was 0.11.1
I had to do
pip install statsmodels=0.12 --force-reinstall

I still get the error with 0.12:
I added an npz (zipped so it would upload)
arr.zip

>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from statsmodels.tsa.stattools import coint
>>> import numpy as np
>>> arr= np.load('arr.npz')
>>> arr = arr['a']
>>> coint( arr[:1024, 0, 5],  arr[:1024, 1, 5],  trend='c', method='aeg', maxlag=10, autolag='t-stat', return_results=Fa
lse)
(-2.0741079140064516, 0.48990605178066554, array([-3.90717771, -3.34210925, -3.04859844]))
>>> coint( arr[:1024, 0, 5],  arr[:1024, 1, 5],  trend='c', method='aeg', maxlag=0, autolag='t-stat', return_results=Fal
se)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 1569, in coint
    res_adf = adfuller(
  File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 309, in adfuller
    icbest, bestlag = _autolag(
  File "C:\Python38\lib\site-packages\statsmodels\tsa\stattools.py", line 143, in _autolag
    return icbest, bestlag
UnboundLocalError: local variable 'bestlag' referenced before assignment

ENV:

>python
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS
------------------
Python: 3.8.2.final.0

statsmodels
===========

Installed: 0.12.0 (C:\Python38\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.19.1 (C:\Python38\lib\site-packages\numpy)
scipy: 1.5.2 (C:\Python38\lib\site-packages\scipy)
pandas: 1.1.1 (C:\Python38\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Python38\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Python38\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.2.0 (C:\Python38\lib\site-packages\matplotlib)
    backend: TkAgg
cvxopt: Not installed
joblib: 0.14.1 (C:\Python38\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.15.0 (C:\Python38\lib\site-packages\IPython)
    jinja2: 2.11.2 (C:\Python38\lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.6.1 (C:\Python38\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed

With random data, coint is fine:

>>> coint( np.random.random(1000),  np.random.random(1000), trend='c', method='aeg', maxlag=0, autolag='t-stat', return_results=False)
(-30.617368909945277, 0.0, array([-3.90743646, -3.34225305, -3.04869817]))

[bashtage]

Still can't reproduce

>>> from statsmodels.tsa.stattools import coint
   ...: >>> import numpy as np
   ...: >>> arr= np.load('arr.npz')
   ...: >>> arr = arr['a']

In [10]: from statsmodels.tsa.api import coint

In [12]: coint( arr[:1024, 0, 5],  arr[:1024, 1, 5],  trend='c', method='aeg', maxlag=10, autolag='t-stat', return_resu
    ...: lts=False)
Out[12]:
(-2.0741079140053276,
 0.4899060517812532,
 array([-3.90717771, -3.34210925, -3.04859844]))

[bashtage]

Ohh, got it

coint( arr[:1024, 0, 5],  arr[:1024, 1, 5],  trend='c', method='aeg', maxlag=0, autolag='t-stat', return_results=False)

[gyllila]

I got the same error message for t-stat, however for adfuller, not coint (I did not use coint). Maybe my fix can help you:

Open the file stattools.py and find the lines (you can first search for which file contains coint):

elif method == “t-stat”:
#stop = stats.norm.ppf(.95)
stop = 1.6448536269514722

Then add here following two lines:
bestlag = startlag
icbest = np.abs(results[startlag].tvalues[-1])

This way, the code won’t crash again and t-stat simply uses no lag when there is no significant value.

What caused this issue? The reason is when using t-stat, bestlag is only assigned, when the last lag becomes significant the first time, so if no lag has a significant t-value, then bestlag is never assigned. I fixed it by first assigning no lag as default to bestlag, then the remaining code would update the values whenever a significant t-value is found.

Hope it helps.

[rjsdotorg]

@gyllila
So, mine
C:\Python38\Lib\site-packages\statsmodels\tsa\stattools.py
now reads:

    elif method == "t-stat":
        # stop = stats.norm.ppf(.95)
        stop = 1.6448536269514722
        bestlag = startlag
        icbest = np.abs(results[startlag].tvalues[-1])
        for lag in range(startlag + maxlag, startlag - 1, -1):
            icbest = np.abs(results[lag].tvalues[-1])
            if np.abs(icbest) >= stop:
                bestlag = lag
                icbest = icbest
                break

The idea being that if np.abs(icbest) is never >= stop it at least has the init value.

[gyllila]

@rjsdotorg

Exactly. Though, this is just a suggestion, you don’t have to do it the same way. In some earlier replies someone mentioned to change autolag, which sounds also to be a good solution, maybe better than mine.

Anyway: does coint now work after adding the two lines?