Skip to content

cnap/smt-for-gec

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 

Systematically Adapting Machine Translation for Grammatical Error Correction

If using this code or its results, please cite

@InProceedings{napoles-callisonburch:2017:BEA,
  author    = {Napoles, Courtney  and  Callison-Burch, Chris},
  title     = {Systematically Adapting Machine Translation for Grammatical Error Correction},
  booktitle = {Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications},
  month     = {September},
  year      = {2017},
  address   = {Copenhagen, Denmark},
  publisher = {Association for Computational Linguistics},
  pages     = {345--356},
  url       = {http://www.aclweb.org/anthology/W17-5039}
}

Requirements

  • python2
  • python libraries:
    • py-enchant (1.6.8)
    • nltk (3.0.5)
    • scipy (0.15.1)
    • inflect (0.2.5)
    • kenlm
    • numpy (1.11.0)
  • Stanford tagger (3.5.0)
  • fast_align
  • Joshua (6.0.5)
  • morpha, morphg (1.0.4)
  • RASP M2 scorer
  • English language model (not supplied)

Set environmental variables JOSHUA, M2_HOME, MORPH_HOME, FASTALIGN, STANFORD_HOME to parent directories of each tool

Instructions

1. Prepare parallel corpus/corpora

To prepare a corpus given two parallel files, PREFIX.err and PREFIX.corr, call

./prepare_corpus.sh PREFIX corpus-name

This will tokenize and tag the texts and generate the token-level alignment. Corpora will be saved in data/corpus-name/

2. Generate morphological analysis

./generate_morpha_lookup.sh -d store data/*/*.vocab

This creates the file store/vocab and store/vocab.morph, a unique list of all of the vocab across in the corpus/corpora and the morphological analysis.

3. Augment grammar with artificial rules

./create_artificial_rules.sh data/corpus-name/ path/to/lm

Generates spelling and morphological rules and saves them to data/corpus-name/[spelling|morph]-rules.gz

4. Calculate GEC-specific features

./calculate_gec_features.sh grammar.gz [grammar1.gz ...]

Calculates features for each grammar and saves the new grammar file to grammar.gec-features.gz

5. Recase and normalize output

python recase.py data/corpus/corpus-name.tok.err.pos candidate.txt >> candidate.cased

Recases lower-case candidate sentences by comparing it to the proper nouns in the sentence

6. Analyze parallel sentences

./compare_edits.sh -i data/corpus/corpus-name.tok.err -c candidate.txt

Performs analysis of sentence pairs and writes to candidate-source.parallel.analysis

Contents

├── README.md
├── data
│   └── test            # final correct test set references
│       └── unprocessed # original test set references
├── results             # system results and Joshua configuration
├── scripts             # system results and Joshua configuration
└── src                 # Joshua customizations

Errata

The results presented in the paper were calculated on an earlier version of the JFLEG test set (commit #) which has since been processed for correcting errors present in the annotations (commit # ). We will be submitting errata to the ACL anthology (stay tuned), and have included both the earlier, the unprocessed test set and the final, processed test set in data/test. The original results were as follows and are incorrect for comparison with other systems reporting results on the JFLEG test set.

System GLEU P R F_0.5
Sp. baseline 55.5 57.7 16.6 38.4
MT baseline 54.8 56.7 14.6 36.0
SMEC+morph 57.9 54.7 44.2 52.3
SMEC-morph 58.3 55.9 41.1 52.2
YB16 58.4 59.4 35.3 52.3
Human 62.1 67.0 52.9 63.6

The newer, correct results are

System GLEU P R F_0.5
Sp. baseline 47.1 58.4 17.4 39.7
MT baseline 45.9 58.4 15.3 37.4
SMEC+morph 54.2 57.4 44.6 54.3
SMEC-morph 53.9 58.2 42.0 54.1
YB16 51.9 60.7 35.5 53.2
Human 62.4 68.8 62.9 67.5

These results should be considered the final results. Systems scored higher when evaluated on the original, unprocessed test set because there were more errors present in that test set and so candidate sentences had greater overlap with those references.

Notes

Stay tuned for our additions to Joshua (namely an adapted GLEU metric for parameter optimization). Feel free to email me if you'd like them now.


Contact napoles@cs.jhu.edu with any questions.

Last updated: 2017-09-07

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published