Simplification Automatic evaluation Measure through Semantic Annotation
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Human_evaluation_benchmark.ods
LICENSE
README.md
SAMSA_abl_score.py
SAMSA_score.py
scene_sentence_alignment.py
scene_sentence_extraction.py

README.md

SAMSA

SAMSA Evaluation Metric and Human Evaluation Benchmark for Text Simplification

If you use the metric or the benchmark, please cite the following paper:

  Semantic Structural Evaluation for Text Simplification
  Elior Sulem, Omri Abend and Ari Rappoport
  Proc. of NAACL 2018

Dataset

./human_evaluation_benchmark.ods

Human evaluation scores given by 5 annotators for the 4 elicitation questions described in the paper. Each annotator scored the same 700 (input,output) pairs.

The source sentences and the system outputs can be found at http://homepages.inf.ed.ac.uk/snaraya2/data/simplification-2016.tgz.

Code

Requirements:

  1. Python NLTK

  2. UCCA toolkit

  3. Monolingual word aligner

  4. The TUPA parser for parsing the source side.

Contents:

./scene_sentence_extraction.py

./scene_sentence_alignment.py

./SAMSA_score.py: The SAMSA metric

./SAMSA_abl_score.py: The SAMSA metric without the non-splitting penalty