Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

Jing Ma¹  Xiang Xiang¹  Ke Wang²  Yuchuan Wu²  Yongbin Li²

¹Huazhong University of Science and Technology 

²Alibaba Group 

This offcial repository contains the code for our CVPR 2024 paper:

Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

For more details, please refer to our paper.

Environment

This implementation is based on PyTorch, and we recommend using a GPU to run the experiments.

Install the required packages using conda and pip:

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt

Run

1. Prepare a Teacher Model

If you don't already have a teacher model ready, you'll need to train one.

python -u project/teacher.py --model ResNet32

# or choose to run in the background monitor and output the logs
nohup python -u project/teacher.py --model ResNet32 > output.log 2>error.log &

2. Deprivatization

The first step is to train a DCGAN to accomplish the deprivatization, as mentioned in our our paper.

python -u project/DCGAN.py --teacher ResNet32

# or choose to run in the background monitor and output the logs
nohup python -u project/DCGAN.py --teacher ResNet32 > output.log 2>error.log &

3. Distillation

The second step carries out black-box knowledge distillation, and you can experiment with different KD methods by changing the {method} parameter.

python -u project/{method}.py --teacher ResNet32 --model ResNet8

# or choose to run in the background monitor and output the logs
nohup python -u project/{method}.py --teacher ResNet32 --model ResNet8 > output.log 2>error.log &

{method} can be one of the following: KD, ML, AL, DKD, DAFL, KN, AM, DB3KD, MEKD.

For MEKD, you can specify the --res_type parameter to choose the response type, which can be: soft, hard.

Citation

If you find our work helpful, please consider citing:

@inproceedings{ma2024aligning,
  title={Aligning Logits Generatively for Principled Black-Box Knowledge Distillation},
  author={Ma, Jing and Xiang, Xiang and Wang, Ke and Wu, Yuchuan and Li, Yongbin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={23148--23157},
  year={2024}
}