Evaluation via Local Installation

The local installation and execution of the evaluation scripts is not recommended since the evaluation tools have been tested in the Docker containers. The recommended method for evaluation is to use a pre-built Docker container instead of your local environment. However, if you wish to install the required dependencies and to attempt to run the evaluation scripts in your local environment, please see instructions for doing so below.

System Requirements

• Install this GitHub repository

git clone https://github.com/UCDenver-ccp/craft-shared-tasks.git

• Install the CRAFT Corpus v3.1.3
  • NOTE: v3.1.3 was used for the 2019 CRAFT Shared Task. More recent versions of CRAFT are available and can be installed in place of v3.1.3 if desired
• 5 GB RAM is required for the Concept Annotation evaluations
• Make sure you have Java >= 8.0
• Install Clojure Boot
• Python v2
• Perl

Specific requirements for the Concept Annotation Task

• Build the BioNLP formatted files to be used as the gold standard annotations
• From the base directory of the CRAFT distribution, run the following (replacing /home/craft/eval-data/concept/bionlp/ with a directory of your choosing):

boot concept -t CHEBI convert -b -o /home/craft/eval-data/concept/bionlp/chebi
boot concept -t CHEBI -x convert -b -o /home/craft/eval-data/concept/bionlp/chebi_ext
boot concept -t CL convert -b -o /home/craft/eval-data/concept/bionlp/cl
boot concept -t CL -x convert -b -o /home/craft/eval-data/concept/bionlp/cl_ext
boot concept -t GO_BP convert -b -o /home/craft/eval-data/concept/bionlp/go_bp
boot concept -t GO_BP -x convert -b -o /home/craft/eval-data/concept/bionlp/go_bp_ext
boot concept -t GO_CC convert -b -o /home/craft/eval-data/concept/bionlp/go_cc
boot concept -t GO_CC -x convert -b -o /home/craft/eval-data/concept/bionlp/go_cc_ext
boot concept -t GO_MF convert -b -o /home/craft/eval-data/concept/bionlp/go_mf
boot concept -t GO_MF -x convert -b -o /home/craft/eval-data/concept/bionlp/go_mf_ext
boot concept -t MOP convert -b -o /home/craft/eval-data/concept/bionlp/mop
boot concept -t MOP -x convert -b -o /home/craft/eval-data/concept/bionlp/mop_ext
boot concept -t NCBITaxon convert -b -o /home/craft/eval-data/concept/bionlp/ncbitaxon
boot concept -t NCBITaxon -x convert -b -o /home/craft/eval-data/concept/bionlp/ncbitaxon_ext
boot concept -t PR convert -b -o /home/craft/eval-data/concept/bionlp/pr
boot concept -t PR -x convert -b -o /home/craft/eval-data/concept/bionlp/pr_ext
boot concept -t SO convert -b -o /home/craft/eval-data/concept/bionlp/so
boot concept -t SO -x convert -b -o /home/craft/eval-data/concept/bionlp/so_ext
boot concept -t UBERON convert -b -o /home/craft/eval-data/concept/bionlp/uberon
boot concept -t UBERON -x convert -b -o /home/craft/eval-data/concept/bionlp/uberon_ext

Specific requirements for the Structural Annotation Task

• Install CoNLL 2018 universal dependency parse evaluation software

wget http://universaldependencies.org/conll18/conll18_ud_eval.py

Specific requirements for the Coreference Resolution Task

• Build the CoNLL 2011/12 formatted gold standard coreference files.
  • From the base directory of the CRAFT distribution, run the following (replacing /home/craft/eval-data/coreference with a directory of your choosing):

boot part-of-speech coreference convert -i -o /home/craft/eval-data/coreference/conllcoref

• Install the coreference evaluation software (requires Perl)

git clone --single-branch --branch lea-integration https://github.com/bill-baumgartner/reference-coreference-scorers.git ./reference-coreference-scorers.git
cd ./reference-coreference-scorers.git
perl test/test.pl

Evaluation of the Concept Annotation Task

After performing the above installation tasks, see Concept Annotation Task Evaluation

Evaluation of the Structural Annotation Task

After performing the above installation tasks, see Structural Annotation Task Evaluation

Evaluation of the Coreference Resolution Task

After performing the above installation tasks, see Coreference Resolution Task Evaluation