.. module:: lenskit.metrics.predict
The :py:mod:`lenskit.metrics.predict` module contains prediction accuracy metrics. These are intended to be used as a part of a Pandas split-apply-combine operation on a data frame that contains both predictions and ratings; for convenience, the :py:func:`lenskit.batch.predict` function will include ratings in the prediction frame when its input user-item pairs contains ratings. So you can perform the following to compute per-user RMSE over some predictions:
preds = predict(algo, pairs) user_rmse = preds.groupby('user').apply(lambda df: rmse(df.prediction, df.rating))
Prediction metric functions take two series, predictions and truth.
.. autofunction:: rmse
.. autofunction:: mae
LensKit rating predictors do not report predictions when their core model is unable
to predict. For example, a nearest-neighbor recommender will not score an item if
it cannot find any suitable neighbors. Following the Pandas convention, these items
are given a score of NaN (when Pandas implements better missing data handling, it will
use that, so use :py:meth:`pandas.Series.isna`/:py:meth:`pandas.Series.notna`, not the
isnan
versions.
However, this causes problems when computing predictive accuracy: recommenders are not being tested on the same set of items. If a recommender only scores the easy items, for example, it could do much better than a recommender that is willing to attempt more difficult items.
A good solution to this is to use a fallback predictor so that every item has a prediction. In LensKit, :py:class:`lenskit.algorithms.basic.Fallback` implements this functionality; it wraps a sequence of recommenders, and for each item, uses the first one that generates a score.
You set it up like this:
cf = ItemItem(20) base = Bias(damping=5) algo = Fallback(cf, base)