This repository consists of a number of Python notebooks and scripts for detecting Leichte Sprache on the web. It is a companion of the following paper:
H. Asghari, F. Hewett, & T. Züger. (2023). "On the Prevalence of Leichte Sprache on the German Web". In Proceedings of WebSci ’23. https://doi.org/10.1145/3578503.3583599
The included notebooks and scripts are as follows:
- 1.training-neural-classifier.ipynb: training notebook for the easy vs. standard German classifier
- 2A.create-our-curlie-set.ipynb: notebook to create curated curlie dataset
- 2B.create-oscar-subset.ipynb: notebook to create subset of Oscar22 domains that are within Curlie and end with .de
- 3.classify-oscar-pages.py: script which applies the classifier to the Oscar subset
- 4.LS-crawler: Scrapy-based crawler to save LeichteSprache pages found on a list of websites
- 5.paper-analysis.ipynb: notebook to replicate our paper's analysis
The following files are additionally included:
- mbow-alldata: classifier model trained on our full dataset (output of step 1)
- dataset-xxx-open.csv: part of the test & train data used to train the classifier (about a quarter of the data that we can share publicly)
- curlie-ourset.csv: our subset of Curlie domains with flattened categories (output of step 2A)
- oscar22-classified.csv.xz: a very large file containing our Oscar22 subset URLs passed through our classifier (output of step 3)
If you have any questions or notice any errors, please feel free to open an issue or contact the authors.