SEVEN：pruning transformer model by reserving sentinels

IJCNN 2024 | [Paper] | [Code]

Our results achieved

TODO List

• Plug-and-Play Implementation of SEVEN
• Experiment: SEVEN_pre on ImageNet
• Experiment: SEVEN_pre on Cifar
• Experiment: SEVEN_dyn on GLUE

Contents

• Clone
• Requirements
• Experiments
• Citation

Clone

You can clone this repo and install it locally.

git clone https://github.com/xiaojinying/SEVEN.git

Requirements

datasets>=2.18.0
easydict>=1.11
transformers>=4.35.2

Experiments

GLUE

To run the SST-2 example with SEVEN, run the following:

python train.py --dataset sst2 --alpha_1 0.8 --alpha_2 0.8 --learning_rate 2e-5 --epoch 10 --batchsize 32 --pruning_algo SEVEN --target_ratio 0.6

Citation

@article{xiao2024seven,
  title={SEVEN: Pruning Transformer Model by Reserving Sentinels},
  author={Xiao, Jinying and Li, Ping and Nie, Jie and Tang, Zhe},
  journal={arXiv preprint arXiv:2403.12688},
  year={2024}
}