Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.

morpheval

Evaluation of the morphological quality of machine translation outputs. The automatically generated test suite in English should be translated into Czech or Latvian. The output is then analyzed and provides three types of information:

  • Adequacy: has the morphological information been well conveyed from the source?
  • Fluency: do we have local agreement?
  • Consistency: how well is the system confident in its prediction?

More details here.

Requirements

How To

Use your MT system to translate the source file morph_test_suite_limsi.en (untokenized). The next steps assume that the outputs are tokenized with Moses tokenizer (use English tokenization for Latvian).

Czech

  • Adequacy and fluency:

    • Run analysis:
      cat morph_test_suite_limsi.translated.cs | sed 's/$/\n/' | tr ' ' '\n' | morphodita/src/run_morpho_analyze czech-morfflex-131112-raw_lemmas.dict 1 --input=vertical --output=vertical > morph_test_suite_limsi.translated.cs.ambig
    • Run evaluation:
      python3 evaluate_morph_pairs_cs.py -i morph_test_suite_limsi.translated.cs.ambig -n morph_test_suite_limsi.en.info
  • Consistency:

    • Run tagging:
      cat morph_test_suite_limsi.translated.cs | sed 's/$/\n/' | tr ' ' '\n' | morphodita/src/run_tagger czech-morfflex-pdt-131112-raw_lemmas.tagger-best_accuracy --input=vertical --output=vertical > morph_test_suite_limsi.translated.cs.disambig
    • Run evaluation:
      python3 evaluate_consistency_cs.py -i morph_test_suite_limsi.translated.cs.disambig -n morph_test_suite_limsi.en.info

Latvian

  • Set the path to LU MII Tagger in tags_lv.sh and run tagging (outputs morph_test_suite_limsi.translated.lv.tag):
    ./tags_lv.sh morph_test_suite_limsi.translated.lv

  • Adequacy and fluency:

    • Generate ambiguities (analysis) with the dictionary:
      python3 make_ambig_lv.py -w morph_test_suite_limsi.translated.lv -t morph_test_suite_limsi.translated.lv.tag > morph_test_suite_limsi.translated.lv.tag.ambig
    • Run evaluation:
      python3 evaluate_morph_pairs_lv.py -i morph_test_suite_limsi.translated.lv.tag.ambig -n morph_test_suite_limsi.en.info
  • Consistency:

    • Format output:
      python3 make_disambig_vert.py -w morph_test_suite_limsi.translated.lv -t morph_test_suite_limsi.translated.lv.tag > morph_test_suite_limsi.translated.lv.tag.disambig
    • Run evaluation:
      python3 evaluate_consistency_lv.py -i morph_test_suite_limsi.translated.lv.tag.disambig -n morph_test_suite_limsi.en.info

Publication

Franck Burlot and François Yvon, Evaluating the morphological competence of machine translation systems. In Proceedings of the Second Conference on Machine Translation (WMT’17). Association for Computational Linguistics, Copenhagen, Denmark, 2017.

About

Evaluation of the morphological quality of machine translation outputs

Resources

Releases

No releases published

Packages

No packages published