ENH: penalization with conjugate priors and artificial observations

[josef-pkt]

Just a thought for a GLM analogue of treating penalties as prior observations.
This is mainly a theoretical issue to see if we get a group of cases where we can have a semi-generic implementation with a consistent theoretical interpretation.

TheilGLS: The original Theil, Goldberger article use the interpretation of the prior as the results from a previous sample. In the normal L2 context this would correspond to a normal-normal conjugate pair (assuming known variance).
The moment conditions just form a weighted mean of prior and sample second moment matrix.

Jeffreys prior for proportion_confint just corresponds to adding 0.5 initial observations to both successes and failures. (beta-binomial conjugate pair)

Firth penalization uses Jeffreys prior more generally for GLM. #3561
I didn't really go through the computational details yet about how they use data augmentation for the Firth case. AFAIU, they change all endog observations directly.

GAM, PIRLS, uses the local WLS approximation to compute locally something similar or analogous to TheilGLS, either adding artificial observations or combining moment matrices.
(The current implementation in GAM adds artificial observations, AFAIR.)

AFAIR, all GLM-families/LEF with canonical link functions have conjugate prior pairs.

maybe this is mostly the point that Greenland makes in several articles I read, reference in #3561

about implementation:

• For cases where we have a (data independent) prior interpretation, we can use penalty functions as in base._penalties
• For data dependent penalization (empirical Bayes and Jeffreys priors) we can try data augmentation or using the likelihood function of the data model to add the penalization.

e.g. maybe, a guess
The analogy to adding 0.5 observations in proportion_confint and in contingency tables could be just adding 0.5 (or something) similar observations for each cell, e.g. if we have empty cells, interaction terms where patsy adds a column of zeros.
I'm not sure how this works for continuous exog, i.e. just start with the one proportion or contingency table case.

[josef-pkt]

#3561 firth penalization