Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
data make linked text consistent with other leaderboard docs Nov 15, 2018
evaluator these are args, not flags Nov 15, 2018
README.md make linked text consistent with other leaderboard docs Nov 15, 2018

README.md

ProPara

Example usage

To evaluate dummy predictions against the ProPara test dataset, run this:

% export PYTHONPATH=.
% evaluator/evaluator.py -p data/test/dummy-predictions.tsv -a data/test/answers.tsv
=================================================
Question     Avg. Precision  Avg. Recall  Avg. F1
-------------------------------------------------
Inputs                1.000        0.241    0.388
Outputs               1.000        0.130    0.230
Conversions           1.000        0.185    0.312
Moves                 1.000        0.222    0.363
-------------------------------------------------
Overall Precision 1.000                          
Overall Recall    0.195                          
Overall F1        0.326                          
=================================================

Evaluated 54 predictions against 54 answers.

Replace data/test/dummy-predictions.tsv with your predictions to compute your test score.

You can also evaluate predictions against the Dev and Training sets by running:

% evaluator/evaluator.py -p data/dev/dummy-predictions.tsv -a data/dev/answers.tsv
% evaluator/evaluator.py -p data/train/dummy-predictions.tsv -a data/train/answers.tsv

See the evaluator documenation for details about the prediction file format and scoring process.