This repository provides the code to easily combines Grammatical Error Correction (GEC) models to produce better predictions with just the models' outputs, as reported in this paper:
Frustratingly Easy System Combination for Grammatical Error Correction
Muhammad Reza Qorib, Seung-Hoon Na, and Hwee Tou Ng
2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) (PDF)
ESC can work with GRECO to produce a more accurate combination. To get the score of each edit, use the argument --score
instead of --test
when combining the base systems' output.
This code should be run with Python 3.6. The reason Python 3.6 is needed is because the ERRANT version that is used in the BEA-2019 shared task (v2.0.0) is not compatible with Python >= 3.7
Install this code dependencies by running:
pip install -r requirements.txt
python -m spacy download en
wget https://www.comp.nus.edu.sg/~nlp/sw/m2scorer.tar.gz
tar -xf m2scorer.tar.gz
Note that you may need to customize your pytorch installation depending on your CUDA version, read more here. The code may also work with torch < 1.9.0 as only simple pytorch functions are used.
For the CoNLL-2014 experiment, run: export EXP_DIR=conll-exp
.
For the BEA-2019 experiment, run: export EXP_DIR=bea-exp
.
- Get the model output
python run.py --test --data_dir $EXP_DIR/test-text --m2_dir $EXP_DIR/test-m2 --model_path $EXP_DIR/models/paper_model.pt --vocab_path $EXP_DIR/paper_vocab.idx --output_path $EXP_DIR/outputs/test.out
- Evaluate the test prediction. Replace test_output with $EXP_DIR/outputs/test.out
For the CoNLL-2014 experiment, run: export EXP_DIR=conll-exp
.
For the BEA-2019 experiment, run: export EXP_DIR=bea-exp
.
- Run the training command:
python run.py --train --data_dir $EXP_DIR/dev-text --m2_dir $EXP_DIR/dev-m2 --model_path $EXP_DIR/models --vocab_path $EXP_DIR/vocab.idx
- Get the prediction on BEA-2019 Dev:
python run.py --test --data_dir $EXP_DIR/dev-text --m2_dir $EXP_DIR/dev-m2 --model_path $EXP_DIR/models/model.pt --vocab_path $EXP_DIR/vocab.idx --output_path $EXP_DIR/outputs/dev.out
- Get the F0.5 development score:
errant_parallel -ori $EXP_DIR/dev-text/source.txt -cor $EXP_DIR/outputs/dev.out -out $EXP_DIR/outputs/dev.m2
errant_compare -ref bea-full-valid.m2 -hyp $EXP_DIR/outputs/dev.m2
- Get the test prediction:
python run.py --test --data_dir $EXP_DIR/test-text --m2_dir $EXP_DIR/test-m2 --model_path $EXP_DIR/models/model.pt --vocab_path $EXP_DIR/vocab.idx --output_path $EXP_DIR/outputs/test.out
- Evaluate the test prediction. Replace test_output with $EXP_DIR/outputs/test.out
- For CoNLL-2014 (requires Python 2.x):
python2 m2scorer/scripts/m2scorer.py test_output conll14st-test-corrected.m2
- For BEA-2019:
Compress the
test_output
into test.zip, then upload the zip file (only containing the prediction file without any folder or any other file) to https://competitions.codalab.org/competitions/20228#participate-get-data
The simplest way is:
- Create a new experiment directory, then go inside this directory.
- Put your base systems' output on BEA-2019 Dev in a folder called
dev-text
. Please also copy thesource.txt
andtarget.txt
from thebea-exp/dev-text
folder to this newdev-text
folder. - Put your base system's output on the test set in a folder called
test-text
. Please also put the source sentences of the dataset you are testing with inside the folder, under the name ofsource.txt
. - Create the
models
andoutputs
folder. At this point, make sure your folder structure is similar to the contents ofbea-exp
orconll-exp
, with the exceptions ofdev-m2
andtest-m2
(The code will generate these folders automatically). - Go back to the parent directory and follow the guide above, with the $EXP_DIR replaced with your new folder name.
If you want to customize your experiment setup, please note:
- The code will index all files in the
--data_dir
folder as base systems, except the source file (the default filename issource.txt
) and the target file (the default filename istarget.txt
). - The code will only read the contents of
--m2_dir
, not--data_dir
. The code will index the files in--data_dir
and look for the file with same basename on the--m2_dir
.If the--m2_dir
does not exist, the code will generate the directory along with the contents from the content of--data_dir
. Thus, if you make any changes to the content of--data_dir
after--m2_dir
was generated, please remove the corresponding file on the--m2_dir
or the delete the whole--m2_dir
entirely. - The file names of the training files and the testing files have to be the same. The file names and the ordering are stored in the vocab file.
- When you run the testing, make sure you run the prediction with the correct model and correct vocab file. Both files are dependent to the base systems you are combining.
The source code and models in this repository are licensed under the GNU General Public License Version 3 (see License). For commercial use of this code and models, separate commercial licensing is also available. Please contact Hwee Tou Ng (nght@comp.nus.edu.sg)