Skip to content
Switch branches/tags

Latest commit

… (#46)

* add a feature to handle zip files as json batches

* Add _create_file_obj

it helps upload both .json and .zip batches

* remove StringIO

Co-authored-by: Hitokazu Matsushita <>

Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Appraise Evaluation Framework

release license: BSD

Appraise is an open-source framework for crowd-based annotation tasks, notably for evaluation of machine translation (MT) outputs. The software is used to run the yearly human evaluation campaigns for shared tasks at the WMT Conference on Machine Translation and other events.

Annotation tasks currently supported in Appraise:

  • Segment-level direct assessment (DA)
  • Document-level direct assessment
  • Pairwise direct assessment (similar to EASL and RankME)
  • Multimodal MT assessment

Getting Started

See for a step-by-step instructions on how to install prerequisites and setup Appraise.


See Examples/ for simple end-to-end examples for setting up currently supported annotation tasks and read how to create your own campaign in Appraise.


This project is licensed under the BSD-3-Clause License.


If you use Appraise in your research, please cite the following paper:

    title = "Appraise Evaluation Framework for Machine Translation",
    author = "Federmann, Christian",
    booktitle = "Proceedings of the 27th International Conference on
        Computational Linguistics: System Demonstrations",
    month = aug,
    year = "2018",
    address = "Santa Fe, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "86--88"