SUMM: influence function for penalized or constrained models

[josef-pkt]

a companion issue to #4257
topic: towards generic local influence functions and measures, more theory and applications
I still don't have a good overview how the different special cases connect.

Here are a few more articles related to recent extensions in statsmodels

The second article by Zhu and Zhang uses the influence function to derive some influence measures. Their setup is based on misspecified likelihood (White 1982)

I haven't read the other papers (AFAIR)

Avella-Medina looks very interesting, e.g. influence function for penalized M-estimators like lasso or scad based on locally smooth approximation. If I understand correctly based on skimming some parts, then the asymptotic variance/distribution only takes the parameters that were not penalized to zero, i.e. block-diagonal where second block is zero. I guess we have a asymptotic justification for using some standard M-estimator sandwich cov_params for the nonzero params, but whether that has good small/medium sized sample properties is not clear to me, and doubtful. (e.g. M-estimator RLM cov_params for a smoothed centile function are not close to the kink, local kernel estimate version in QuantReg.)
The local smoothing is pretty much what I used in penalized/penalties which is the computational trick used by Fan and others.

Paula and Cysneiros's article is closer to linear model and GLM and just has a bunch of linear algebra similar to the score_test for linear constraints. (based only a 1 minute look)

Venezuela et al I didn't look at it, but it also needs to handle the sandwich form for the covariance and is not based on an optimization problem, i.e. GEE case.

There are more related articles where I didn't look much, e.g. on influence in linear Ridge regression, and of course a very large number for specific distributions (but that's mostly standard MLE)

References

Avella-Medina, Marco. 2017. “Influence Functions for Penalized M-Estimators.” Bernoulli 23 (4B): 3178–96. https://doi.org/10.3150/16-BEJ841.

Zhu, Hongtu, and Heping Zhang. 2004. “A Diagnostic Procedure Based on Local Influence.” Biometrika 91 (3): 579–89. https://doi.org/10.1093/biomet/91.3.579.

Paula, Gilberto A., and Francisco José A. Cysneiros. 2010. “Local Influence Under Parameter Constraints.” Communications in Statistics - Theory and Methods 39 (7): 1212–28. https://doi.org/10.1080/03610920902871438.

Venezuela, Maria Kelly, Mônica Carneiro Sandoval, and Denise Aparecida Botter. 2011. “Local Influence in Estimating Equations.” Computational Statistics & Data Analysis 55 (4): 1867–83. https://doi.org/10.1016/j.csda.2010.10.020.

[josef-pkt]

another one, looks easy to implement, essentially generalized Ridge (TheilGLS with iid observations).
setup is linear model with L2 penalized spline component, i.e. sounds useful for GAM (without the G)
They provide formulas for partial influence functions, i.e. separating influence on linear parametric and spline/nonparametric parts. (which looks like influence of one part after "absorbing" the other part)
(I read only first few pages)

Zhu, Zhong-Yi, Xuming He, and Wing-Kam Fung. 2003. “Local Influence Analysis for Penalized Gaussian Likelihood Estimators in Partially Linear Models.” Scandinavian Journal of Statistics 30 (4): 767–80. https://doi.org/10.1111/1467-9469.00363.