Skip to content

Structural Annotation Task Evaluation

Jump to bottom
Bill Baumgartner edited this page May 1, 2020 · 13 revisions

Docker image version

Please see the README for information on which Docker image [VERSION] to use.

Expected Input Format

The Structural Annotation evaluation takes dependency parse files in the CoNLL-U file format as input. Many thanks to Manuel Ciosici and Sampo Pyysalo for their help with generating and vetting the CoNLL-U files for the CRAFT dependency parses. See examples here.

Expected Directory Structure

The evaluation platform expects a single directory containing one CoNLL-U file per document. Files must use the same identifier as used by the source text file. For example, the dependency parse file derived from 11532192.txt should be named 11532192.conllu.

Running an Evaluation via Docker (Recommended)

Assuming files generated by your system containing dependency parses in the CoNLL-U format for the documents in the CRAFT corpus are in the following directory: /local/path/to/system/output/, use the following command to evaluate your system against CRAFT:

docker run --rm -v /local/path/to/system/output:/files-to-evaluate ucdenverccp/craft-eval:[VERSION] sh -c 'cd /home/craft/evaluation && boot eval-univ-dependency'

Evaluation results will be written to a file in the /local/path/to/system/output directory. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted.

Sanity checking your results by self-evaluation via Docker

CRAFT Shared Task participants have been asked to validate their results prior to result submission by evaluating their results against themselves. This section demonstrates how to do that using the available Docker container.

Required directories:

  • /local/path/to/system/output/ - path to the system output files (your result submission)

To run the self-evaluation:

docker run --rm -v /local/path/to/system/output:/files-to-evaluate ucdenverccp/craft-eval:[VERSION] sh -c '(cd /home/craft/evaluation && boot eval-univ-dependency -b /home/craft/evaluation -s /home/craft/evaluation/universal-dependency -i /files-to-evaluate -g /files-to-evaluate)'

Evaluation results will be written to a file in each of the system output directories. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted. Check to ensure that the evaluation completed successfully, and that F-score is 1.0 prior to submitting your results.

Running an Evaluation using a Local Installation

Assuming all required dependencies are installed on your local system as described here, the following command will evaluate your system output against CRAFT (when run from the base directory of the the project available in this GitHub repository):

boot eval-dependency -b /path/to/evaluation/directory -i /path/to/system/output -g /path/to/CRAFT/structural-annotation/dependency/conllu -s /path/to/directory/containing/conll18_ud_eval.py

where:

  • /path/to/evaluation/directory is the local path to the evaluation/ directory of the project in this very repository
  • /path/to/CRAFT is the path to the CRAFT distribution base directory
  • /path/to/system/output is the path to the directory containing dependency parse files (one for each document) in the CoNLL-U file format that were produced by the system being evaluated.
  • /path/to/directory/containing/conll18_ud_eval.py is the path to the directory containing the conll18_ud_eval.py script.

Evaluation results will be written to a file in the /path/to/system/output directory.

Clone this wiki locally