Mixed Effects/Hierarchical GLM support

[exalate-issue-sync]

No description provided.

[exalate-issue-sync]

Wendy commented: In mixed-effect GLM, the response variable is a function of x^TB+zgi^Tbi+ei for group i. This means all groups share the common coefficients B. For each subgroup, it has a set of coefficients bi belong to that group. Hence, we need to estimate the coefficients that are common to all groups and also the coefficients bi for each group.

At a first glance, this will involve a lot of data movement. I need to group the data as a whole unit to estimate B, then, into many subgroups to estimate bi. This only involves a bunch of calls to manipulate the data Frame.

As usual, we need to derive the loglikelihood, get the gradient/hessian, and try to solve for how to change the coefficients. This is not going to be trivial at all because we have one global coefficient sets and several local coefficients.

[exalate-issue-sync]

Wendy commented: Information from customers:

Example analysis with open data: [https://ourcodingclub.github.io/2017/03/15/mixed-models.html#two|https://ourcodingclub.github.io/2017/03/15/mixed-models.html#two]

Example implementation in R: [https://cran.r-project.org/web/packages/lme4/index.html|https://cran.r-project.org/web/packages/lme4/index.html]

Example implementation in a distributed environment: [https://github.com/linkedin/photon-ml|https://github.com/linkedin/photon-ml|smart-link]

[exalate-issue-sync]

Wendy commented: There are several distribution combinations for HGLM and they are:

• family = gaussian, rand.family = Gaussian
• family = binomial, rand.family = beta
• family = binomial, rand.family = aaussian
• family = poisson, rand.family = gaussian
• family = poisson, rand.family = aamma
• family = gamma, rand.family = gaussian
• family = gamma, rand.family=gamma
• family = gamma, rand.family = inverse.gamma

However, learning from Michalk, I am going to break each implementation into a JIRA.

[exalate-issue-sync]

William Prucknic commented: Could tweedie be included?

[exalate-issue-sync]

Wendy commented: William: Tweedie be added to family or rand.family? Please let me know. Wendy

[exalate-issue-sync]

William Prucknic commented: I noticed tweedie missing from the list of family's above and when using GLMs I use Tweedie often when analyzing insurance data. That’s why I made the suggestion.

However, I don’t have any experience w/ mixed effects models, so I know I am out of my element here.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-6509
Assignee: Wendy
Reporter: Nidhi Mehta
State: Resolved
Fix Version: 3.28.0.1
Attachments: N/A
Development PRs: N/A