TextFooler

A Model for Natural Language Attack on Text Classification and Inference

This is the source code for the paper: Jin, Di, et al. "Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment." arXiv preprint arXiv:1907.11932 (2019). If you use the code, please cite the paper:

@article{jin2019bert,
  title={Is bert really robust? natural language attack on text classification and entailment},
  author={Jin, Di and Jin, Zhijing and Zhou, Joey Tianyi and Szolovits, Peter},
  journal={arXiv preprint arXiv:1907.11932},
  year={2019}
}

Prerequisites:

• Pytorch >= 0.4
• Tensorflow >= 1.0
• Numpy
• Python >= 3.6

How to use

• Run the following code to install the esim package:

cd ESIM
python setup.py install
cd ..

• (Optional) Run the following code to pre-compute the cosine similarity scores between word pairs based on the counter-fitting word embeddings.

python comp_cos_sim_mat.py [PATH_TO_COUNTER_FITTING_WORD_EMBEDDINGS]

• Run the following code to generate the adversaries for text classification:

python attack_classification.py

For Natural langauge inference:

python attack_nli.py

Examples of run code for these two files are in run_attack_classification.py and run_attack_nli.py. Here we explain each required argument in details:

• --dataset_path: The path to the dataset. We put the 1000 examples for each dataset we used in the paper in the folder data.
• --target_model: Name of the target model such as ''bert''.
• --target_model_path: The path to the trained parameters of the target model. For ease of replication, we shared the trained BERT model parameters on each dataset we used.
• --counter_fitting_embeddings_path: The path to the counter-fitting word embeddings.
• --counter_fitting_cos_sim_path: This is optional. If given, then the pre-computed cosine similarity scores based on the counter-fitting word embeddings will be loaded to save time. If not, it will be calculated.
• --USE_cache_path: The path to save the USE model file (Downloading is automatic if this path is empty).