fix bug in loo and p_loo computation, also return waic and loo result as dataframes #1765
Conversation
pymc3/stats.py
Outdated
| elif pointwise: | ||
| return waic, waic_se, waic_i, p_waic | ||
| if pointwise: | ||
| return pd.DataFrame([[waic, waic_se, p_waic, waic_i]], |
There was a problem hiding this comment.
This should be a pd.Series I think (or maybe a namedtuple).
There was a problem hiding this comment.
Yeah, I think namedtuple would be best here as we also use it for e.g. ADVI.
There was a problem hiding this comment.
I thought the same, but Jupyter renders DataFrame in a nicer way. And I am also thinking this is more consistent with a compare function (that I need to write) that will display waic/loo results for several models as a DataFrame.
There was a problem hiding this comment.
@twiecki still thinking namedtuple is the way to go? If so let me now and I will change it.
There was a problem hiding this comment.
Your argument is purely for display purposes? I'm not convinced that should guide API and data-structure choices.
There was a problem hiding this comment.
It's possible to convert, but a bit clunky:
from collections import namedtuple
Point = namedtuple('test', ['x', 'y'])
p = Point(1, 2)
pd.Series(p._asdict()).to_frame()There was a problem hiding this comment.
yep purely aesthetics. OK. I will use namedtuples. Thanks for the review.
pymc3/stats.py
Outdated
| w = np.minimum(r_new, r_new.mean(axis=0) * S**0.75) | ||
|
|
||
| loo_lppd_i = -2.0 * logsumexp(log_py, axis = 0, b = w / np.sum(w, axis = 0)) | ||
| loo_lppd_i = - 2. * logsumexp(log_py, axis=0, b=w/np.sum(w, axis=0)) |
|
Thanks @aloctavodia! |
While trying to address what we discuss in #1677 I found that a previous PR introduced an error in the computation of
p_loo(a variable was undefined). This PR fix it. I also found a bug in the computation of lppd_loo related to the unsorting of the importance ratios. Now the results ofpm.looare closer to those reported for the 8 school problem in the papers "Understanding predictive information criteria for Bayesian models" and "Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC".