Using Improper Priors

[bhgomes]

I am wondering how Turing deals or plans to deal with improper priors (especially non-normalizable priors like uniform over unbounded domains) in its estimation of the posterior distribution. I haven't seen anything in the documentation describing usage or best practices. In my attempts, I have not been able to get good estimates (naively) using improper priors, i.e. bad convergence, vanishing derivatives, ... etc.

[yebai]

@bhgomes Thanks for the question. Please play with the flat distribution (#315 (comment)) and see whether it fits the need here. Feel free to post your findings here.

[bhgomes]

I've looked at Flat (and FlatPos) and my issue with them is that they sample a Uniform(0, 1) distribution (and claim to have [-Inf, Inf] coverage) which is not correct. In this case, since they are not normalizable, no quantile function exists. These kinds of "priors" should be interpreted as "don't sample from the prior just sample from the likelihood". I have yet to study the internals of the Turing samplers so I don't know how this would be done in practice, but at least in the case of Flat/FlatPos the situation is clear: just don't sample from the prior, but use the support of these "distributions" as constraints on your integration.

[xukai92]

You can override the rand for Flat to any coverage you want. This would help for particle based-samplers but for HMC it doesn't matter, as the rand funtion is hardly used (I say hardly because the only place you may use is for initilisation. But even for that Turing.jl use the Stan way for initialisation.).

So step back to the point of impropr priors. Flat is purely an example to show how to do that by youself. For HMC or MH, this kind of implementation is enough, as they only work on the unnormalised posterior (controlled by logpdf) but not even sampling (rand).

So what other impropr priors do you have in mind? Maybe we can look into the implementaion concretly.