Repository navigation
Structural Annotation Task Evaluation
SA Task evaluation requires files in the CoNLL-X annotation format. Assuming files for the all documents in the CRAFT corpus are in the following directory: /local/path/to/system/output/, use the following command to evaluate your system against CRAFT:
docker run --rm -v /local/path/to/system/output:/files-to-evaluate ccp/craft-eval:3.1.1_0 sh -c 'boot eval-dependency-parses'
Dependency parse evaluation requires files in the CoNLL-U annotation format. They should be named using the document id conjoined with the .conllu suffix, e.g. 1532192.conllu. The gold standard files are available in the CRAFT v3.1 distribution in structural-annotation/dependency/conllu.
Assuming files to be evaluated (one for each document in the CRAFT corpus) are in the following directory: /path/to/system/output/, use the following command to evaluate your system's dependency parses against CRAFT:
cd /path/to/repository/download/evaluation (Note, this is the directory containing the build.boot file.)
boot eval-dependency -b /path/to/repository/download -i /path/to/system/output -g /path/to/CRAFT/download/structural-annotation/dependency/conllu -s /path/to/directory/containing/conll18_ud_eval.py
boot eval-dependency -b /Users/bill/Dropbox/work/projects/craft-shared-task-2019/craft-st-eval-platform.git/evaluation -i /Users/bill/Dropbox/work/projects/craft-shared-task-2019/sample-test-data/dependency -g /Users/bill/Dropbox/work/projects/craft-shared-task-2019/CRAFT.bill.git/structural-annotation/dependency/conllx -s /Users/bill/Dropbox/work/projects/craft-shared-task-2019/sa-task/malt-eval/dist-20141005