Concept Annotation Task Evaluation

Docker image version

Please see the README for information on which Docker image [VERSION] to use.

Expected Input Format

CA Task evaluation requires files in the BioNLP stand-off annotation format. The evaluation code assumes that ontology concepts are stated as the annotation type, and are not connected to an annotation using the 'Normalization' N lines. e.g. an annotation for the CL concept for 'rod photoreceptors' (CL:0000604) should look like the following:

T3      CL:0000604 30862 30865;30875 30889      rod ... photoreceptors

Note, that the example above is a discontinuous annotation comprised of two separate spans.

Evaluating the extension class annotations

The CRAFT corpus contains some supplementary resources which may be beneficial, in particular, when evaluating using the extension class annotations. Please see information detailing these supplementary resources here.

Expected Directory Structure

For this task, annotations will be evaluated on a per-ontology basis, and files should be organized in directories named by the ontology for which they use. For example, submitting the following directory structure (rooted in the concept_system_output/ directory) to the evaluation platform would result in evaluations of annotations for the CL, CL_EXT, and UBERON ontologies:

concept_system_output/
concept_system_output/cl
concept_system_output/cl_ext
concept_system_output/UBERON

The ontology keys, i.e. the names of the directories, must be one of the following:

CHEBI
CHEBI_EXT
CL
CL_EXT
GO_BP
GO_BP_EXT
GO_CC
GO_CC_EXT
GO_MF
GO_MF_EXT
MONDO_WITHOUT_GENOTYPE_ANNOTATIONS
MONDO_WITH_GENOTYPE_ANNOTATIONS
MOP
MOP_EXT
NCBITAXON
NCBITAXON_EXT
PR
PR_EXT
SO
SO_EXT
UBERON
UBERON_EXT

The _EXT suffix is used to specify the ontology+extension classes should be used for the evaluation. Note that MONDO does not have extension class annotations. For MONDO there are two choices, the MONDO_WITH_GENOTYPE_ANNOTATIONS and the MONDO_WITHOUT_GENOTYPE_ANNOTATIONS annotation sets.

Note that the code treats the directory names case-insensitively, so cl and CL are both valid directory names for storing annotations to be evaluated against the CRAFT CL gold standard. Files must use the same identifier as used by the source text file. For example, a concept annotation file derived from 11532192.txt should be named 11532192.bionlp.

Running an evaluation via Docker (recommended)

Assuming directories for the all document sets to be evaluated are in the following directory: /local/path/to/system/output/, and are named appropriately as described above, the following command will evaluate your system output against CRAFT:

docker run --rm -v /local/path/to/system/output:/files-to-evaluate ucdenverccp/craft-eval:[VERSION] sh -c '(cd /home/craft/evaluation && boot javac eval-concept-annotations)'

Evaluation results will be written to a file in each of the system output directories. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted.

Sanity checking your results by self-evaluation via Docker

CRAFT Shared Task participants have been asked to validate their results prior to result submission by evaluating their results against themselves. This section demonstrates how to do that using the available Docker container.

Required directories:

• /local/path/to/system/output/ - path to the system output files (your result submission)
• /local/path/to/corpus/distribution/ - path to the base directory of the corpus distribution. In the case of the CRAFT shared task evaluation, this should be the base directory of the test data project. For the CA task, this directory must have an articles/txt/ directory containing the plain text versions of the corpus documents and a concept-annotation directory containing the relevant ontology files organized in the same structure as the CRAFT corpus.

To run the self-evaluation:

docker run --rm -v /local/path/to/system/output:/files-to-evaluate -v /local/path/to/corpus/distribution:/corpus-distribution ucdenverccp/craft-eval:[VERSION] sh -c '(cd /home/craft/evaluation && boot javac eval-concept-annotations -c /corpus-distribution -i /files-to-evaluate -g /files-to-evaluate)'

Evaluation results will be written to a file in each of the system output directories. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted. Check to ensure that the evaluation completed successfully, and that F-score is 1.0 prior to submitting your results.

Running an evaluation using a local installation

Generate the gold standard files in the BioNLP format

The first step to running an evaluation using your local installation is to generate the concept annotation files to be used as the gold standard. The evaluation platform requires these files to use the BioNLP format, and the CRAFT distribution is capable of generating these files. To generate them, use the following command (again, from base directory of the CRAFT distribution, not from the Boot script in this project):

boot concept -t CHEBI convert -b -o /path/to/gold-standard/bionlp/chebi
boot concept -t CHEBI -x convert -b -o /path/to/gold-standard/bionlp/chebi_ext
boot concept -t CL convert -b -o /path/to/gold-standard/bionlp/cl
boot concept -t CL -x convert -b -o /path/to/gold-standard/bionlp/cl_ext
boot concept -t GO_BP convert -b -o /path/to/gold-standard/bionlp/go_bp
boot concept -t GO_BP -x convert -b -o /path/to/gold-standard/bionlp/go_bp_ext
boot concept -t GO_CC convert -b -o /path/to/gold-standard/bionlp/go_cc
boot concept -t GO_CC -x convert -b -o /path/to/gold-standard/bionlp/go_cc_ext
boot concept -t GO_MF convert -b -o /path/to/gold-standard/bionlp/go_mf
boot concept -t GO_MF -x convert -b -o /path/to/gold-standard/bionlp/go_mf_ext
boot concept -t MONDO convert -b -o /path/to/gold-standard/bionlp/mondo
boot concept -t MONDO -x convert -b -o /path/to/gold-standard/bionlp/mondo_ext
boot concept -t MOP convert -b -o /path/to/gold-standard/bionlp/mop
boot concept -t MOP -x convert -b -o /path/to/gold-standard/bionlp/mop_ext
boot concept -t NCBITaxon convert -b -o /path/to/gold-standard/bionlp/ncbitaxon
boot concept -t NCBITaxon -x convert -b -o /path/to/gold-standard/bionlp/ncbitaxon_ext
boot concept -t PR convert -b -o /path/to/gold-standard/bionlp/pr
boot concept -t PR -x convert -b -o /path/to/gold-standard/bionlp/pr_ext
boot concept -t SO convert -b -o /path/to/gold-standard/bionlp/so
boot concept -t SO -x convert -b -o /path/to/gold-standard/bionlp/so_ext
boot concept -t UBERON convert -b -o /path/to/gold-standard/bionlp/uberon
boot concept -t UBERON -x convert -b -o /path/to/gold-standard/bionlp/uberon_ext

where:

• /path/to/gold-standard/ is the local (absolute) path where you would like the gold standard BioNLP-formatted files to be located

Evaluate your system output

Assuming all required dependencies are installed on your local system as described here, the following command will evaluate your system output against CRAFT (when run from the base directory of the the project available in this GitHub repository):

boot javac eval-concept-annotations -c /path/to/CRAFT -g /path/to/gold/standard/bionlp/base/directory -i /path/to/system/output/bionlp/base/directory

where:

• /path/to/CRAFT is the path to the CRAFT distribution base directory
• /path/to/gold/standard/bionlp/base/directory is the path to the base directory containing ontology-specific directories of BioNLP formatted gold standard files. These files were created during the local installation process.
• /path/to/system/output/bionlp/base/directory is the path to the base directory containing ontology-specific directories of BioNLP formatted files produced by the system to be evaluated.

Evaluation results will be written to a file in each of the system output directories.