In the recent years, transformer-based language learning models like BERT have been one of the most popular architectures used in the research related to natural lan- guage processing. Productionizing these models under constrained resources such as edge computing require smaller versions of BERT like DistilBERT. However, it is a concern that these models have inadequate robustness against adversarial examples and attacks. This paper evaluates the performance of various models built on the ideas of adversarial training and GAN BERT finetuned on SST-2 dataset. Further the experiments in this paper seek to find evidence on whether knowledege distillation preserves robustness in the student models.
Vijay Kalmath Department of Data Science vsk2123@columbia.edu
Amrutha Varshini Sundar Department of Data Science as6431@columbia.edu
Sean Chen Department of Computer Science sean.chen@columbia.edu
Google Drive Link to Models : https://drive.google.com/drive/folders/1q1TGTOl6BftZzn1ZidzvN8GKVRbTdrjM?usp=sharing