Evaluation via Docker (Recommended Method)

Docker image version

Please see the README for information on which Docker image [VERSION] to use.

Evaluation

Evaluating your system output using the pre-built Docker container is the recommended approach as the Docker container has all dependencies installed, including the CRAFT corpus.

For those of you using a Windows system, use of the Docker container is highly encouraged as the evaluation tools have not been tested on Windows.

System requirements

• Install Docker
• 5 GB RAM is required for the Concept Annotation evaluations

Once Docker is installed, you can evaluate your system output using the commands detailed below. The Docker container installed with the evaluation tools will be downloaded from DockerHub the first time any of those commands are invoked.

Evaluation of the Concept Annotation Task

See Concept Annotation Task Evaluation

Evaluation of the Structural Annotation Task

See Structural Annotation Task Evaluation

Evaluation of the Coreference Resolution Task

See Coreference Resolution Task Evaluation