SkillRet Evaluation Metrics

SkillRet Evaluation Metrics

SkillRet metrics measure how well retrieved skills match a query's labeled needs. @K means the top K retrieved skills, in ranked order. Qrels (ground-truth relevance judgments) assign relevance grades to query–skill pairs; positive grades identify relevant skills. These scores evaluate retrieval, not successful skill execution.

Metrics comparison

Scores range from 0 to 1; higher is better. Dataset scores average the per-query values.

Metric	Full name	What it measures	Interpretation
NDCG@K	Normalized Discounted Cumulative Gain	Relevance near the top, compared with an ideal ranking. Supports graded relevance.	1 means ideal ranking quality at K.
Recall@K	Recall	Fraction of all relevant skills retrieved within K.	1 means every relevant skill was found.
Precision@K	Precision	Fraction of the K result slots containing relevant skills.	Higher means fewer irrelevant results.
MAP@K	Mean Average Precision	Averages precision at relevant-hit ranks, then averages across queries.	Rewards finding more relevant skills earlier.
Completeness@K	Completeness	Whether every required, relevant skill appears within K.	Per query: 1 if complete, otherwise 0; the mean is the complete-query rate.
MRR@K	Mean Reciprocal Rank	Averages the reciprocal rank of the first relevant result within K.	1 means the first result is relevant; no hit within K scores 0.

The evaluators use relevance grades directly as NDCG gain, with logarithmic rank discounts. Precision divides by K even for shorter lists. Average Precision (AP@K) divides the sum of precision at relevant-hit ranks by all relevant skills, including missed ones.

One query, top five

Required skills: A, B, C, each with relevance grade 1. Retrieved order: X, A, Y, C, Z; X, Y, Z are irrelevant.

Metric	Value	Reason
NDCG@5	≈ 0.498	Hits at ranks 2 and 4 versus ideal ranks 1, 2, 3.
Recall@5	2/3 ≈ 0.667	Two of three relevant skills found.
Precision@5	2/5 = 0.400	Two relevant results in five slots.
MAP@5	(1/2 + 2/4)/3 ≈ 0.333	AP@5; MAP equals AP for this single query.
Completeness@5	0	B is missing.
MRR@5	1/2 = 0.500	First relevant result is at rank 2.

Official versus project metrics

Verified against the linked source snapshots:

Implementation	Metrics computed	K values
Official SkillRet trec_eval	NDCG, Recall, Precision, MAP, Completeness; no MRR.	Defaults: 5, 10, 15; custom cutoffs through k_values.
MIA5150 fusion evaluator	All five official metrics, plus MRR.	Five official metrics: 5, 10, 15; MRR: 5, 10.

Completeness is already official. Other project studies use different cutoffs; consult the benchmark catalog. MIA5150's fusion evaluator also counts missing/empty rankings as zero across all qrels queries; the official function averages the queries returned by pytrec_eval.

Recommended project metrics

Metric	Why use it?
Recall@5	Measures relevant skill coverage within a small retrieval budget.
NDCG@10	Measures ranking quality, rewarding relevant skills placed earlier.
Completeness@5	Checks whether every required skill was retrieved; partial coverage can leave a task unfinished.
MRR@5	Measures how quickly the first relevant skill appears when scanning results.