SUMM: singular exog (again)

[josef-pkt]

mainly for linear models, but some might also affect other models.

I'm looking at singular exog again in context of pivoting QR.
I don't have an overview of open issues and possible design.

related

• positive definite hessian in optimization TST: Add test for PD hessian #3099
  • AFAIR, no check if hessian is computed in fit and skip_hessian is in optimization
• discussion on not raising if singular Design: OLS issue, perfect collinearity #3824
• a FAQ issue for ill-conditioning SUMM/ENH/FAQ avoiding data problems from ill conditioning and tools #6381
• asking for dropping instead of pinv Singular design matrix produce very extreme p-values #8670
• PR for pivoting QR ENH: add functionality Improve the stability for regression models with multicollinearity to statsmodels.regression (… #6935
• a similar issue to this one ENH/SUMM/Review (multi-) collinearity checks and warnings for cov_params  #7695
• old issue for Logit, this should now raise Logit returns nan p-value for some coefficients #3367
• MixedLM singular, I'm not familiar with this Linear Mixed Effects Models: numpy.linalg.LinAlgError: Singular matrix #6140

to do, to check:

• cutoffs, thresholds: OLS, ... need an option in fit for rcond in pinv REF: pinv threshold, linear regression for almost singular #2628
  • it looks like matrix rank of cov_param can be much smaller than rank of exog. I guess because of squaring
  • if we use different linalg computations, then we need to check that matrix rank definitions, i.e. thresholds, are consistent, e.g. pinv rcond versus np.linalg.matrix_rank. Our rank attribute in OLS might not be the same as the pinv rank and not the same as the rank of cov_params. pinv rcond and multicollinearity, different cutoffs  #1662 REF/SUMM: numpy lstsq changes default rcond in 1.14 #3994
  • need to report/warn rank of cov_params if not full rank. But constrained estimation, `fit_constrained, has intentionally reduced rank cov_params, variance is zero of constraints cov_params reduced rank - consequences #1717
  • pinv threshold in OLS is too small, many collinear cases slip through.
• Where is the function that I wrote to drop variables sequentially based on QR? I cannot find it right now.
  https://github.com/statsmodels/statsmodels/pull/2380/files#diff-6c9338f4faac471ff45da570aa82c563661ada8aff75afe24e70f60ae9b0ea85R586 there is also an iterative version somewhere (not mine)
• scipy.linalg pivoting QR looks pretty "random", that is in repeated runs with different random samples, which variable is dropped varies.
• empty cells, columns with only zeros (this is more urgent) patsy does not check whether cells are empty when creating the design matrix for categorical interactions. AFAIR pinv works to set those params to zero. but df for anova_lm and wald tests can be wrong. SUMM/FAQ: empty cells, unobserved effects, categorical interactions #4137 BUG/ENH: singular exog and anova_lm #5398 BUG: Nested ANOVA incorrect results -- wrong matrix definition #8506 BUG/ENH: constrained/collinear fit and degrees of freedom for terms, df_terms, anova #8512
• postestimation, hypothesis tests with singular cov_params. Not clear, not systematically checked, (maybe relevant BUG: GLM score_test singular matrix #4798)
• still no "estimable" checks and helper functions. ENH: add is_estimable check for linear hypothesis, linear constraints #6271
• fit_collinear ENH: fit_collinear, unit tests and make public, advertise #8580
• ...