terminology_evaluation

Installation and Prerequisites

The script uses Python 3. You can simply run the following to clone this repository and install all of the above requirements:

git clone https://github.com/mahfuzibnalam/terminology_evaluation.git
cd terminology_evaluation
pip install -r requirements.txt

List of requirements:

1. stanza
2. argparse
3. sacrebleu
4. bs4
5. lxml (need it for mac)

Code

The main script is evaluate_term_wmt.py that receives the following arguments:

1. --language - The language code (eg. fr for French) of the target language.
2. --hypothesis - This is the hypothesis file. Example file: data/en-fr.dev.txt.truecased.sgm.
3. --source - This is a file with the source references. An example file is provided at data/dev.en-fr.en.sgm
4. --target_reference - This is a file with the target references. An example file is provided at data/dev.en-fr.fr.sgm
5. --BLEU [True/False]. By default True. If True shows BLEU score.
6. --EXACT_MATCH [True/False]. By default True. If True shows Exact Match score.
7. --WINDOW_OVERLAP [True/False]. By default True. If True shows Window Overlap Score.
8. --MOD_TER [True/False]. By default True. If True shows TERm score.
9. --TER [True/False]. By default False. If True shows TER score.

Example

You can test that your metrics work by running the following command on the sample data we provide.

python3 evaluate_term_wmt.py \
    --language fr \
    --hypothesis data/en-fr.dev.txt.truecased.sgm \
    --source data/dev.en-fr.en.sgm \
    --target_reference data/dev.en-fr.fr.sgm

Running the above command will:

• Download the French Stanza models, if they are not available locally already
• Compute four metrics and print the following:

BLEU score: 45.33867641150976
Exact-Match Statistics
        Total correct: 759
        Total wrong: 127
        Total correct (lemma): 15
        Total wrong (lemma): 0
Exact-Match Accuracy: 0.8590455049944506
Window Overlap Accuracy :
        Window 2:
        Exact Window Overlap Accuracy: 0.29693757867032844
        Window 3:
        Exact Window Overlap Accuracy: 0.2907071747339513
1 - TERm Score: 0.5976316319523398

Notes:

• The computation of TER or TERm can take quite some time if your data has very long sentences.

Publications

Please cite this papers:

@article{DBLP:journals/corr/abs-2106-11891,
  author    = {Md Mahfuz Ibn Alam and
               Antonios Anastasopoulos and
               Laurent Besacier and
               James Cross and
               Matthias Gall{\'{e}} and
               Philipp Koehn and
               Vassilina Nikoulina},
  title     = {On the Evaluation of Machine Translation for Terminology Consistency},
  journal   = {CoRR},
  volume    = {abs/2106.11891},
  year      = {2021},
  url       = {https://arxiv.org/abs/2106.11891},
  eprinttype = {arXiv},
  eprint    = {2106.11891},
  timestamp = {Wed, 30 Jun 2021 16:14:10 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2106-11891.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}