Skip to content


Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Source code for our ACL2020 paper: On the Robustness of Language Encoders against Grammatical Errors


Python >= 3.5

Download and install berkeleyparser.

Install python requirments via requirments file: pip install -r requirements.txt


Download datasets

The General Language Understanding Evaluation (GLUE) benchmark aims to analyze model ability in natural language understanding. We use some tasks of GLUE as downstream tasks.

You should follow the instructions in this repo to download GLUE benchmark and unpack it to your $data_dir.

The CoNLL-2014 Shared Task: Grammatical Error Correction is where we collect error statistics.

Follow the instructions in this page to download NUCLE Release3.2 and annotated test data.

Remember to change the file path in line 13 and 141 of utils/ to your own path.

Download pre-trained models

For experiments regarding Infersent, you need to download fastText embeddings and the corresponding pre-trained Infersent model.

curl -Lo
curl -Lo examples/infersent2.pkl


Downstream task evaluations

The framework in this repo allows evaluating BERT, RoBERTa, and Infersent on MRPC, MNLI, QNLI, and SST-2. We will provide an example of evaluating bert-base-uncased on MRPC dataset.

The follows can be done with bash . But let's elaborate it step by step. The script for Infersent is .

First train or fine-tune models on clean data (${data_dir} indicates where you store data):

python --mode fine-tune --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

To inject grammatical errors using adversarial attack algorithms, you need to assign the importance score to each token (not necessary for genetic algorithm):

python --mode score --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

then, run the attack algorithms, --adv_type can be greedy, beam_search or genetic:

python --mode attack  --adv_type greedy --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

To inject grammatical errors based on berkeleyparser (the probabilistic case in our paper), you need to first obtain the syntactic parse tree for each sentence in the dataset. Then run:

python csv --input_tsv ${data_dir}/MRPC/dev.tsv --parsed_sent1 ${data_dir}/MRPC/parsed_sent1 --parsed_sent2 ${data_dir}/MRPC/parsed_sent2 --output_tsv ${data_dir}/MRPC/mrpc.tsv --rate 0.15

then, test the model under the probabilistic case:

python --mode attack --adv_type random --random_attack_file ${data_dir}/MRPC/mrpc.tsv --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC 

Note that our framework is flexible. If you want to test new models, you can simply add a new class in like what we did ( See for detials, the new class mainly tells attack algorithms how to construct and forward a new instance with the tested model):

class infersent_enc(object):
    def __init__(self, infersent, config):
        self.infersent = infersent
        self.config = config

    def make_instance(self, text_a, text_b, label, label_map):
        sent1s = [' '.join(text_a)]
        if isinstance(text_b, list):
            sent2s = [' '.join(text_b)]
            sent2s = [text_b]
        return [sent1s, sent2s, [label]]

    def model_forward(self, model, batch):
        sent1s, sent2s, label_ids = [list(item) for item in batch]
        sent1_tensor = self.infersent.encode(sent1s, tokenize=True)
        sent2_tensor = self.infersent.encode(sent2s, tokenize=True)
        return logits

Model layer evaluations

Navigate to the probing directory:

cd probing



The input data contains three columns seperated by /t, where the first column indicates the data split tr, va or te; the second column indicates the binary correctness of this example 0, 1; the third column is the sentence. For example:

tr      1       It knows we want to eat a doughnut , not drink it .

BERT masked language model evaluations

We provide some examples for a quick check.

Navigate to the probing directory:

cd probing



To collect the complete data, navigate to the examples directory and run:

python --type *error_type*

where *error_type* can be Prep, ArtOrDet, Wci, Trans, Nn, SVA, Vform.


Our framework is developed based on PyTorch implementations of BERT and RoBERTa from PyTorch-Transformers, Infersent from SentEval, and ELMo from AllenNLP and Jiant.

We also borrowed and edited code from the following repos: nlp_adversarial_examples, nmt_grammar_noise, interpret_bert.

We would like to thank the authors of these repos for their efforts.


If you find our work useful, please cite our ACL2020 paper: On the Robustness of Language Encoders against Grammatical Errors

  author = {Yin, Fan and Long, Quanyu and Meng, Tao and Chang, Kai-Wei},
  title = {On the Robustness of Language Encoders against Grammatical Errors},
  booktitle = {ACL},
  year = {2020}


Source code for ACL2020: On the Robustness of Language Encoders against Grammatical Errors







No releases published


No packages published