Untold Story of Preprocessing Task Evaluation

Alignment-based Joint Evaluation Algorithm for the Preprocessing Task

This folder contains the code for the paper.

Requirements

conda create --name <env> --file requirements.txt

Data

The data is available at "ud-treebanks-v2.xx", which can be downloaded from here. We use this data to generate all the data we need.

Data Generation

Run the following command one by one to generate the data we need.

python src/nltk_gen.py
python src/stanza_gen.py
python src/conllu2txt.py

Note: The src/conllu2txt.py should only be executed after the src/nltk_gen.py and src/stanza_gen.py are executed, and can only be executed once. If you want to re-generate the data, please delete the data/txt-gold and data/txt-stanza folders first.

Algorithm

The algorithm is implemented in the src/jp_algorithm.py

Evaluation

src/evaluate.ipynb

Citation

Eunkyul Leah Jo, Angela Yoonseo Park, Grace Tianjiao Zhang, Izia Xiaoxiao Wang, Junrui Wang, MingJia Mao, and Jungyeul Park. 2024. An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1327–1338, Torino, Italy. ELRA and ICCL.

@inproceedings{jo-etal-2024-untold-story,
    title = "An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach",
    author = "Jo, Eunkyul Leah  and
      Park, Angela Yoonseo  and
      Zhang, Grace Tianjiao  and
      Wang, Izia Xiaoxiao  and
      Wang, Junrui  and
      Mao, MingJia  and
      Park, Jungyeul",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italy",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.119",
    pages = "1327--1338",
}