ENH: More informative error message in fit_regularized Hessian inversion fail

[jseabold]

See #659.

[nfick]

I attempted a fix and created a pull request, but I have not figured out how to test my fix...I am working on that.

[kshitij10496]

@jseabold
Is this issue still open ?

[josef-pkt]

I don't remember any changes, but running the example from #659 works now

>>> print(Logit(yes, data).fit_regularized().summary())
Optimization terminated successfully.    (Exit mode 0)
            Current function value: 1.12892750712e-10
            Iterations: 35
            Function evaluations: 35
            Gradient evaluations: 35
                           Logit Regression Results                           
==============================================================================
Dep. Variable:                      y   No. Observations:                    4
Model:                          Logit   Df Residuals:                        1
Method:                           MLE   Df Model:                            2
Date:                Mon, 07 Dec 2015   Pseudo R-squ.:                   1.000
Time:                        14:08:29   Log-Likelihood:            -4.5157e-10
converged:                       True   LL-Null:                       -2.7726
                                        LLR p-value:                   0.06250
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
const         11.2901   4.47e+12   2.53e-12      1.000   -8.75e+12    8.75e+12
x1           -45.2747   1.02e+05     -0.000      1.000      -2e+05       2e+05
x2            11.2901   4.47e+12   2.53e-12      1.000   -8.75e+12    8.75e+12
==============================================================================

Complete Separation: The results show that there iscomplete separation.
In this case the Maximum Likelihood Estimator does not exist and the parameters
are not identified.

It prints the Separation Warning in the summary but doesn't raise or warn during optimization.

[josef-pkt]

After trying a few alpha it still fails, which is weird because it doesn't fail for alpha=0.1

>>> print(Logit(yes, data).fit_regularized(alpha=0.5).summary())
Optimization terminated successfully.    (Exit mode 0)
            Current function value: 0.578411041679
            Iterations: 12
            Function evaluations: 12
            Gradient evaluations: 12
Traceback (most recent call last):
  File "<pyshell#214>", line 1, in <module>
    print(Logit(yes, data).fit_regularized(alpha=0.5).summary())
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\discrete\discrete_model.py", line 443, in fit_regularized
    size_trim_tol=size_trim_tol, qc_tol=qc_tol, **kwargs)
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\discrete\discrete_model.py", line 357, in fit_regularized
    cov_params_func=cov_params_func, **kwargs)
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\base\model.py", line 437, in fit
    Hinv = cov_params_func(self, xopt, retvals)
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\discrete\discrete_model.py", line 377, in cov_params_func_l1
    H_restricted_inv = np.linalg.inv(-H_restricted)
  File "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\linalg\linalg.py", line 520, in inv
    ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
  File "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\linalg\linalg.py", line 90, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.linalg.LinAlgError: Singular matrix

[devendrakushwah80]

Hi, I’d like to work on this issue by improving the documentation and
error message clarity around Hessian inversion failures in fit_regularized.

I am not planning to change the optimization logic, but to clarify why this
error can occur (e.g. separation, collinearity, regularization strength)
and what users can try to resolve it.

Please let me know if this approach sounds good.