-
Notifications
You must be signed in to change notification settings - Fork 0
Coreference Resolution Task Evaluation
Please see the README for information on which Docker image [VERSION] to use.
The Coreference Annotation evaluation takes files in the CoNLL 2011/12 coreference file format as input. The file format has been augmented to support annotations with discontinuous spans as members of identity chains.
Discontinuous mentions are denoted by the addition of a character or characters (non-digit) after the chain identifier (integer). In the example below, all spans denoted by 32a belong to the same discontinuous mention that is a member of identity chain #32. The next discontinuous mention in identity chain #32 (if there is one), would be denoted by 32b. There is no constraint on the number of characters that can be used to identify a mention, e.g. 32aaaaaaa is perfectly valid. The only constraint is that the only integer in the mention identifier must apply to the identity chain, and that the integer must occur at the beginning of the identifier string, e.g. aaaaaa32 is not a valid mention identifier, nor is 32abc123.
There are three mentions in the sentence below extracted from document PMC194730 (PMID: 12925238). Each mention is a member of a different identity chain. Coreference information is located in the final column of this data file format.
-
"The protein"
- a contiguous mention that starts at token 1 and ends at token 2 (inclusive)
- a member of identity chain #2
-
"a variable N-terminal .. domain"
- a discontinuous mention that includes tokens [16-20] and token 27
- a member of identity chain #32
-
"a conserved C-terminal domain"
- a contiguous mention that includes tokens [22-27]
- a member of identity chain #33
194730 0 1 The DT - - - - - - - (2
194730 0 2 protein NN - - - - - - - 2)
194730 0 3 belongs VBZ - - - - - - - -
194730 0 4 to IN - - - - - - - -
194730 0 5 a DT - - - - - - - -
194730 0 6 family NN - - - - - - - -
194730 0 7 of IN - - - - - - - -
194730 0 8 evolutionarily RB - - - - - - - -
194730 0 9 conserved VBN - - - - - - - -
194730 0 10 proteins NN - - - - - - - -
194730 0 11 of IN - - - - - - - -
194730 0 12 a DT - - - - - - - -
194730 0 13 bipartite JJ - - - - - - - -
194730 0 14 structure NN - - - - - - - -
194730 0 15 with IN - - - - - - - -
194730 0 16 a DT - - - - - - - (32a
194730 0 17 variable JJ - - - - - - - -
194730 0 18 N NN - - - - - - - -
194730 0 19 - HYPH - - - - - - - -
194730 0 20 terminal JJ - - - - - - - 32a)
194730 0 21 and CC - - - - - - - -
194730 0 22 a DT - - - - - - - (33
194730 0 23 conserved VBN - - - - - - - -
194730 0 24 C NN - - - - - - - -
194730 0 25 - HYPH - - - - - - - -
194730 0 26 terminal JJ - - - - - - - -
194730 0 27 domain NN - - - - - - - (32a)|33)
194730 0 28 . . - - - - - - - -
The evaluation platform expects a single directory containing one CoNLL 2011/12 coreference file per document. Files must use the same identifier as used by the source text file. For example, the coreference file derived from 11532192.txt should be named 11532192.conll.
Assuming files generated by your system containing coreference chains in the CoNLL 2011/12 coreference format are in the following directory: /local/path/to/system/output/, use the following command to evaluate your system against CRAFT:
docker run --rm -v /local/path/to/system/output:/files-to-evaluate ucdenverccp/craft-eval:[VERSION] sh -c 'cd /home/craft/evaluation && boot eval-coreference'
Evaluation results will be written to a file in the /local/path/to/system/output directory. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted.
CRAFT Shared Task participants have been asked to validate their results prior to result submission by evaluating their results against themselves. This section demonstrates how to do that using the available Docker container.
Required directories:
-
/local/path/to/system/output/- path to the system output files (your result submission) -
/local/path/to/corpus/distribution/- path to the base directory of the corpus distribution. In the case of the CRAFT shared task evaluation, this should be the base directory of the test data project. For the CR task, this directory must have anarticles/txt/directory containing the plain text versions of the corpus documents.
To run the self-evaluation:
docker run --rm -v /local/path/to/system/output:/files-to-evaluate -v /local/path/to/corpus/distribution:/corpus-distribution ucdenverccp/craft-eval:[VERSION] sh -c '(cd /home/craft/evaluation && boot eval-coreference -c /corpus-distribution -i /files-to-evaluate -g /files-to-evaluate -b /home/craft/evaluation -s /home/craft/evaluation/coreference/reference-coreference-scorers.git)'
Evaluation results will be written to a file in the system output directory. Note: The Docker container cannot write to an encrypted filesystem, however, so please make sure /local/path/to/system/output references a directory that is not encrypted. Check to ensure that the evaluation completed successfully, and that F-score is 1.0 prior to submitting your results.
The first step to running an evaluation using your local installation is to generate the coreference annotation files in the CoNLL-Coref 2011/12 file format to be used as the gold standard. The CRAFT distribution is capable of generating this file format. To do so, use the following command (again, from base directory of the CRAFT distribution, not from the Boot script in this project):
boot part-of-speech coreference convert --conll-coref-ident -o /path/to/gold-standard/coref/
where:
-
/path/to/gold-standard/coref/is the local (absolute) path where you would like the gold standard CoNLL-Coref 2011/12 formatted files to be located
Note that it is necessary to include the part-of-speech annotations as the CoNLL-Coref 2011/12 format requires token and sentence boundaries.
Assuming all required dependencies are installed on your local system as described here, the following command will evaluate your system output against CRAFT (when run from the base directory of the the project available in this GitHub repository):
boot eval-coreference -b /path/to/evaluation/directory -i /path/to/system/output -g /path/to/gold/conllcoref2011-12/files -s /path/to/coreference/scoring/github/repo -c /path/to/CRAFT
where:
-
/path/to/evaluation/directoryis the local path to theevaluation/directory of the project in this very repository -
/path/to/CRAFTis the path to the CRAFT distribution base directory -
/path/to/system/outputis the path to the directory containing coreference annotation files (one for each document) in the CoNLL 2011/12 coreference file format that were produced by the system being evaluated. -
/path/to/gold/conllcoref2011-12/filesis the path to the directory of gold standard CoNLL Coreference 2011/12 formatted files. These files were created as part of the local installation procedure for this task. -
/path/to/coreference/scoring/github/repois the path to the scoring code which can be found in a GitHub project that was downloaded as part of the local installation procedure for this task.
Evaluation results will be written to a file in the /path/to/system/output directory.