Metrics dcg score

Development build. This page describes main, not a released package. The latest published Lodestar.Metrics is 0.3.0 — read its documentation.

Dcg.Score

The mean discounted gain over the rows — sklearn.metrics.dcg_score.

public static double Score(ReadOnlySpan<double> yTrue, ReadOnlySpan<double> yScore, int labelCount, int? k = null, double logBase = 2, bool ignoreTies = false, ReadOnlySpan<double> sampleWeight = default)

Parameters — yTrue is the relevance of each document and yScore the scores the ranking was made from, both row-major: one row per query, labelCount values each, and the same length. k scores only the first k positions, or null for all of them; a k past labelCount scores the whole row rather than raising. logBase is the base of the positional discount, 2 as in scikit-learn, and anywhere in (0, ∞): a base below 1 is accepted and takes the score negative, 1 itself makes every discount zero. ignoreTies ranks equal scores in descending index order instead of averaging over their permutations — faster, and correct only when genuine ties cannot occur. sampleWeight carries one weight per query, or is empty for an unweighted mean; over a single query it cancels, since it multiplies both halves of the mean.

Returns — double, the mean over the rows of Σ relevance / log(rank + 1). Unbounded above: it grows with the relevance values, so two rows are comparable only on the same judgement scale. Use Ndcg.Score for a number in [0, 1].

Exceptions — ArgumentException when labelCount is below 2 (scikit-learn's own sentence, "Computing NDCG is only meaningful when there is more than 1 document."), when yTrue and yScore disagree in length, or when the length is not a whole number of rows of labelCount. ArgumentOutOfRangeException when k is below 1, and when logBase falls outside (0, ∞) — zero, negative, NaN or infinite; scikit-learn refuses the same values, through the constraint it prints as "must be a float in the range (0.0, inf)". A zero or negative base would otherwise reach the caller as a silent NaN score. ArgumentException also when sampleWeight is neither empty nor one value per query, or when it sums to zero — numpy.average's own refusal, which the reference reaches as a ZeroDivisionError from the same call.

A negative relevance is not refused here, and the result can be negative — dcg_score accepts it too. Ndcg.Score does refuse it, because there the ratio would leave [0, 1].

Example — four documents whose scores are all equal, scored both ways.

using Lodestar.Metrics;

double[] relevance = [3, 2, 1, 0];
double[] tied = [0.5, 0.5, 0.5, 0.5];

double averaged = Dcg.Score(relevance, tied, labelCount: 4);  // => 3.8424…
double arbitrary = Dcg.Score(relevance, tied, labelCount: 4, ignoreTies: true);  // => 2.9229…

Remarks — the gains are linear. Much of the literature uses 2^relevance − 1 instead, which on the row above ranked perfectly gives 9.3927… where this gives 4.7618…; the difference is the definition, not an error on either side. The averaged and arbitrary values in the example differ by almost a third, which is the whole reason ignoreTies defaults to false.

Applies to — net10.0, netstandard2.0.

See also — Ndcg.Score, ReciprocalRank.Score, the Python equivalence table.