[DOC] add an exmaple for LASSO

[stevenlis]

Please consider adding an example for LASSO regression.

Btw, I came across the following two resources, which might be useful for the implementation of selective inference.
https://www.pnas.org/content/pnas/112/25/7629.full.pdf
https://selective-inference.github.io/Python-software/

[kshedden]

An example would be welcome as a PR. Eventually I may get to it but not in the near future.

Thanks for the link regarding selective inference methods. It would be nice to include some of these. Note that we do already have the Knockoff Filter, which is also a method for selective inference.

Our Lasso is implemented generically, and works for most models (not just for linear regression), e.g. it works for all GLM's and survival analysis. We also have a separate approach to doing selective inference with GEE. I'm not sure how general these particular selective inference methods that you cited are.

[stevenlis]

@kshedden Thanks for the info. Sounds like the SM implement could be quite useful, which exactly why we need examples. Since when setting lambda = 0 and it just becomes a simple OLS, and simple example for OLS would be nice.

[emilmirzayev]

I volunteer. Anyone can assign me on this ?

@kshedden

[bashtage]

Thanks.

[emilmirzayev]

Looks like #3670 is still open. I can use result.params as a workaround as suggested

[stevenlis]

but the result.params doesn't give you standard error right? If so, is there any way for inference right now?

Btw, do we have cross-validation implemented in the model? or just use a sklearn to pickle a lambda and then fit a lasso in SM to get more statistics like standard error for inference?

What a general workflow for Lasso with SM now?

[josef-pkt]

We don't have inference for Lasso.

AFAIR, it is possible to estimate an unpenalized OLS after Lasso with the selected variables. That should have the inference, BUT it ignores the variable selection problem. (which is not really that much different from a specification search by the user or stepwise regression which is also ignore inference.)

The old discrete fit_regularized has inference for the nonzero parameters but they also ignore the variable selection problem.
In penalized maximum likelihood, we added inference which is appropriate for smooth penalization like L2, but does not properly take penalization to zero into account with L1 penalization or kinks at zero.
a new stackoverflow question also has two related references https://stackoverflow.com/questions/58577753/prediction-intervals-for-penalized-linear-regression

[kshedden]

Standard error doesn't mean much here since the limiting distribution is far from Gaussian.

[josef-pkt]

I think standard inference should be good enough for large coefficients in large samples.
I don't remember details right now, but that's what I remember from Jianqing Fan's articles on SCAD and sure independence screening. (asymptotically we also have correct variable selection)