Warn in low-dimensional cases

[wwiecek]

If N = 2 we should consider stopping people from meta-analysing
If N = 3 we should suggest a prior for scale (half-Cauchy)

That means we also need to implement half-Cauchy

[wwiecek]

Prior for scale parameters in hierarchical models
Gelman (2006) suggested half-Cauchy with mode at 0 and scale set to a large value (in the 8-schools example, we used the value 25), or with the scale estimated from data in a hierarchical-hierarchical setting in which there are many variance parameters which can be given a common prior.
The Gelman (2006) recommendations may be too weak for many purposes. If the number of groups is small, the data don't provide much information on the group-level variance, and so it can make sense to use stronger prior information, in two ways. First: Cauchy might be too broad, maybe better to use something like a t_4 or even half-normal if you don't think there's a chance of any really big values. Second: maybe the scale parameter for this hyperprior should be set to something reasonable, not to something large. This would suggest something like half-normal(0,1) or half-t(4,0,1) as default choices.
Historically, a prior on the scale parameter with a long right tail has been considered "conservative" in that it allows for large values of the scale parameter which in turn correspond to minimal pooling. But from a modern point of view, minimal pooling is not a default, and a statistical method that underpools can be thought of as overreacting to noise and thus "anti-conservative."
If doing modal estimation, see section on Boundary Avoiding Priors above

[be-green]

RE: N = 2, I don't know if this is a good option, but Andrew Gelman discusses multilevel models with 2 groups here and argues that it's fine but you need an informative prior on the between-group sd. Maybe warning about using an informative prior if there are only two groups is appropriate?

I think it's also hard to determine what "informative" means in advance of seeing the data, since whether a prior is informative probably depends on the context. Anyway just thought I'd mention it.

[wwiecek]

The problem is more about A) users who don't know what they're doing supplying data that is not formatted correctly (so N=2 is just a glitch), B) trying to set a prior automatically based on just 2 data points (for now the default prior is scaled with regards to dispersion in inputs). We can be extra careful on B), but nevertheless there should be a warning.

…

On Tue, Aug 20, 2019 at 5:29 PM be-green ***@***.***> wrote: RE: N = 2, I don't know if this is a good option, but Andrew Gelman discusses multilevel models with 2 groups here <https://statmodeling.stat.columbia.edu/2015/12/08/hierarchical-modeling-when-you-have-only-2-groups-i-still-think-its-a-good-idea-you-just-need-an-informative-prior-on-the-group-level-variation/> and argues that it's fine but you need an informative prior on the between-group sd. Maybe warning about using an informative prior if there are only two groups is appropriate? I think it's also hard to determine what "informative" means in advance of seeing the data, since whether a prior is informative probably depends on the context. Anyway just thought I'd mention it. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=AE5QPJTDRROC5QMVJMIBJGLQFQLV3A5CNFSM4GPPE4DKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4W4COQ#issuecomment-523092282>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5QPJU2TXSYYTDTXVGQBJDQFQLV3ANCNFSM4GPPE4DA> .

[wwiecek]

There is now a warning for small N, but the default prior is unchanged -- worth revisiting this and setting a different prior (see text above)

[wwiecek]

We now have automated warning on setting default priors for N=2, N=3, but we do not stop anyone.