Skip to content

Commit

Permalink
Improve metric documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
mdekstrand committed Nov 6, 2018
1 parent 95cb90f commit f468355
Show file tree
Hide file tree
Showing 2 changed files with 57 additions and 1 deletion.
41 changes: 41 additions & 0 deletions doc/evaluation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,44 @@ data set, you can do::
# group and aggregate
nbr_ndcg = user_ndcg.groupby('max_neighbors').nDCG.mean()
nbr_ndcg.plot()

Prediction Accuracy Metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. module:: lenskit.metrics.predict

The :py:mod:`lenskit.metrics.predict` module containins prediction accuracy metrics.

.. autofunction:: rmse
.. autofunction:: mae

Top-*N* Accuracy Metrics
~~~~~~~~~~~~~~~~~~~~~~~~

.. module:: lenskit.metrics.topn

The :py:mod:`lenskit.metrics.topn` module contains metrics for evaluating top-*N*
recommendation lists.

Classification Metrics
----------------------

These metrics treat the recommendation list as a classification of relevant items.

.. autofunction:: precision
.. autofunction:: recall

Ranked List Metrics
-------------------

These metrics treat the recommendation list as a ranked list of items that may or may not
be relevant.

.. autofunction:: recip_rank

Utility Metrics
---------------

The nDCG function estimates a utility score for a ranked list of recommendations.

.. autofunction:: ndcg
17 changes: 16 additions & 1 deletion lenskit/metrics/topn.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ def recall(recs, relevant):
Args:
recs(array-like): a sequence of recommended items
relevant(set-like): the set of relevant items
Returns:
double: the fraction of relevant items that were recommended.
"""
check.check_value(not isinstance(relevant, set),
"set type not supported for relevant set",
Expand All @@ -52,10 +55,14 @@ def recall(recs, relevant):
def recip_rank(recs, relevant):
"""
Compute the reciprocal rank of the first relevant item in a recommendation list.
This is used to compute MRR.
Args:
recs(array-like): a sequence of recommended items
relevant(set-like): the set of relevant items
Return:
double: the reciprocal rank of the first relevant item.
"""
check.check_value(not isinstance(relevant, set),
"set type not supported for relevant set",
Expand All @@ -80,6 +87,14 @@ def ndcg(scores, items=None, discount=np.log2):
Compute the Normalized Discounted Cumulative Gain of a series of scores. These should be
relevance scores; they can be :math:`{0,1}` for binary relevance data.
Discounted cumultative gain is computed as:
.. math::
\\begin{align*}
\\mathrm{DCG}(L,u) & = \\sum_{i=1}^{|L|} \\frac{r_{ui}}{d(i)} & \\\\
\\mathrm{nDCG}(L, u) & = \\frac{\\mathrm{DCG}(L,u)}{\\mathrm{DCG}(L_{\\mathrm{ideal}}, u)}
\\end{align*}
Args:
scores(pd.Series or array-like):
relevance scores for items. If ``items`` is ``None``, these should be in order
Expand All @@ -92,7 +107,7 @@ def ndcg(scores, items=None, discount=np.log2):
if the discount is greater than 1.
Returns:
The nDCG of the scored items.
double: the nDCG of the scored items.
"""

if not isinstance(scores, pd.Series):
Expand Down

0 comments on commit f468355

Please sign in to comment.