This is the evaluation script for the FIRE Shared Task on Trannsliterated Search Subtask II - Mixed Script Adhoc Retrieval.
This script evaluates the NDCG@k, MRR, MAP and recall for the supplied runs according to the relevance judgements (qrels). The script is developed for the purpose to evaluate the runs submitted to Shared Task on Transliterated Search at FIRE 2014 but can be used outside it as well.
perl msir14-eval.pl <qrel-file> <run-file> <verbose-level>
perl msir14-eval.pl sample_qrel.txt sample_run.txt 0
Query NDCG@1 NDCG@5 NDCG@10 MAP MRR RECALL average 1.0000 0.7871 0.8502 0.7083 1.0000 0.8750
There are three verboser levels [0-2].
- 0 - It prints the evaluation measures averaged over all queries only.
- 1 - It prints the evaluation measures for each query and average.
- 2 - It prints the gold ranklist and the your ranklist for each query in addtion to the info printed in level-1.
Format of files
- sample_run.txt (single-space delimited):
<query-id> Q0 <docid> <rank> <similarity-score>
- sample_qrel.txt (single-space delimited):
<query-id> QO <docid> <relevance-score>
where, relevance scores are between [0-4] ranging from irrelevant=0 to perfectly relevant=4.
How to get
Either you can fork this github repository or you can simply download the repository as ZIP by clicking "Download ZIP" button.
Parth Gupta, email: email@example.com
Copyright (C) 2012, 2013, 2014, 2015 Parth Gupta.