Evaluation of the morphological quality of machine translation outputs
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
utils
README.md
evaluate_consistency_cs.py
evaluate_consistency_lv.py
evaluate_morph_pairs_cs.py
evaluate_morph_pairs_lv.py
make_ambig_lv.py
make_disambig_vert.py
tags_lv.sh

README.md

morpheval

Evaluation of the morphological quality of machine translation outputs. The automatically generated test suite in English should be translated into Czech or Latvian. The output is then analyzed and provides three types of information:

  • Adequacy: has the morphological information been well conveyed from the source?
  • Fluency: do we have local agreement?
  • Consistency: how well is the system confident in its prediction?

More details here.

Requirements

How To

Use your MT system to translate the source file morph_test_suite_limsi.en (untokenized). The next steps assume that the outputs are tokenized with Moses tokenizer (use English tokenization for Latvian).

Czech

  • Adequacy and fluency:

    • Run analysis:
      cat morph_test_suite_limsi.translated.cs | sed 's/$/\n/' | tr ' ' '\n' | morphodita/src/run_morpho_analyze czech-morfflex-131112-raw_lemmas.dict 1 --input=vertical --output=vertical > morph_test_suite_limsi.translated.cs.ambig
    • Run evaluation:
      python3 evaluate_morph_pairs_cs.py -i morph_test_suite_limsi.translated.cs.ambig -n morph_test_suite_limsi.en.info
  • Consistency:

    • Run tagging:
      cat morph_test_suite_limsi.translated.cs | sed 's/$/\n/' | tr ' ' '\n' | morphodita/src/run_tagger czech-morfflex-pdt-131112-raw_lemmas.tagger-best_accuracy --input=vertical --output=vertical > morph_test_suite_limsi.translated.cs.disambig
    • Run evaluation:
      python3 evaluate_consistency_cs.py -i morph_test_suite_limsi.translated.cs.disambig -n morph_test_suite_limsi.en.info

Latvian

  • Set the path to LU MII Tagger in tags_lv.sh and run tagging (outputs morph_test_suite_limsi.translated.lv.tag):
    ./tags_lv.sh morph_test_suite_limsi.translated.lv

  • Adequacy and fluency:

    • Generate ambiguities (analysis) with the dictionary:
      python3 make_ambig_lv.py -w morph_test_suite_limsi.translated.lv -t morph_test_suite_limsi.translated.lv.tag > morph_test_suite_limsi.translated.lv.tag.ambig
    • Run evaluation:
      python3 evaluate_morph_pairs_lv.py -i morph_test_suite_limsi.translated.lv.tag.ambig -n morph_test_suite_limsi.en.info
  • Consistency:

    • Format output:
      python3 make_disambig_vert.py -w morph_test_suite_limsi.translated.lv -t morph_test_suite_limsi.translated.lv.tag > morph_test_suite_limsi.translated.lv.tag.disambig
    • Run evaluation:
      python3 evaluate_consistency_lv.py -i morph_test_suite_limsi.translated.lv.tag.disambig -n morph_test_suite_limsi.en.info

Publication

Franck Burlot and François Yvon, Evaluating the morphological competence of machine translation systems. In Proceedings of the Second Conference on Machine Translation (WMT’17). Association for Computational Linguistics, Copenhagen, Denmark, 2017.