Skip to content

huji-nlp/SAMSA

 
 

Repository files navigation

SAMSA

SAMSA Evaluation Metric and Human Evaluation Benchmark for Text Simplification

If you use the metric or the benchmark, please cite the following paper:

  Semantic Structural Evaluation for Text Simplification
  Elior Sulem, Omri Abend and Ari Rappoport
  Proc. of NAACL 2018

Dataset

./human_evaluation_benchmark.ods

Human evaluation scores given by 5 annotators for the 4 elicitation questions described in the paper. Each annotator scored the same 700 (input,output) pairs.

The source sentences and the system outputs can be found at http://homepages.inf.ed.ac.uk/snaraya2/data/simplification-2016.tgz.

Code

Requirements:

  1. Python NLTK

  2. UCCA toolkit

  3. Monolingual word aligner

  4. The TUPA parser for parsing the source side.

Contents:

./scene_sentence_extraction.py

./scene_sentence_alignment.py

./SAMSA_score.py: The SAMSA metric

./SAMSA_abl_score.py: The SAMSA metric without the non-splitting penalty

About

Simplification Automatic evaluation Measure through Semantic Annotation

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%