Added output for pointwise predictive accuracy in waic.

[gladomat]

In order to estimate the difference in the predictive accuracy of two models using waic you need the pointwise predictive accuracy. These values can also be used to compute the standard error of the difference. I only changed what is to be returned.

For more on how to do this check out the pymc3 implementation of Chapter 7.5 from McElreath 2016.

Edit: Or better yet, check out the compare.R function from the stan-dev repository.

[aloctavodia]

LGTM. BTW I think we should also have a compare function in PyMC3. @twiecki any thoughts on this?

[twiecki]

Perhaps we should return a dictionary or NamedTuple instead containing all the variables?

@aloctavodia what would that compare function do?

[aloctavodia]

or maybe a Pandas' DataFrame, is like a dict but the output looks nicer. Or this will be too much?

[aloctavodia]

@twiecki the compare function is used to compare two or more models based on their WAIC values. The idea is that you use WAIC and the standard error of WAIC. The function also computes weights for each model (based on WAIC) to be used for model averaging and also the standard error of the difference in WAIC of each model against the top ranked model.

McElreath computes the weights in his book, although the people from the Stan team do not compute them anymore because they think the weights are not very reliable given that their computation ignores the uncertainty. I think we could compute them but adding a warning.

[gladomat]

So who's willing to implement it?

[aloctavodia]

@gladomat probably that should be a separated PR. If you want to do it and have the time you can do it. If not I offer myself, just let me know.

[gladomat]

I won't have any time this week. I think I'll pass on this one. It shouldn't be too hard, especially with the stan code available.

[twiecki]

@aloctavodia doesn't seem like a high-prize item then.

[twiecki]

Probably pandas Series would be the right thing. I think that makes sense. Should we do the same for the other metrics?

[gladomat]

@twiecki not really, it's more of a convenience: give it two models and let it tell me which one seems best.

[aloctavodia]

Loo and Waic should return the same output (as aSeries). The rest of the metrics (dic and bic) returns just one value, as implemented now (and should probably remain that way).

[twiecki]

@aloctavodia Sounds good, want to add that?

[aloctavodia]

@twiecki sure. I will merge this PR and then I will continue from here ASAP.

[aloctavodia]

Thanks @gladomat!