In this repository, we explore model compression for transformer architectures via quantization. We specifically explore quantization aware training of the linear layers and demonstrate the performance for 8 bits, 4 bits, 2 bits and 1 bit (binary) quantization.
[ICML 2022] This work investigates the compatibility between label smoothing (LS) and knowledge distillation (KD). We suggest to use an LS-trained teacher with a low-temperature transfer to render high performance students.
Add a description, image, and links to the iwslt topic page so that developers can more easily learn about it.
To associate your repository with the iwslt topic, visit your repo's landing page and select "manage topics."