OKDPH

This repository contains the code for CVPR2023 OKDPH： Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation.

Data

3 datasets were used in the paper:

• CIFAR-10
• CIFAR-100
• ImageNet: Downloadable from https://image-net.org/download.php

For downloaded data sets please place them in the 'dataset' folder.

dataset:

-- cifar-10-batches-py

-- cifar-100-python

Requirements

• PyTorch 1.0 or higher
• Python 3.6 or higher

Run

cd src
bash OKDPH.sh

For the case of four students:

cd src
python OKDPH.py --omega 0.8 --beta 0.8 --gamma 0.5 --interval 1_epoch \
    --model_names resnet32 resnet32 resnet32 resnet32 \
    --transes hflip cutout augment auto_aug base \
    --log 21_cifar10_okdph_4stu_1ep

Please refer to the bash files for more running commands.

Baselines

cd src
bash baseline.sh

Experiment

• Loss Landscape
• Sample Limited Data

Citation

If you find this work useful for your research, please cite our paper:

@inproceedings{zhang2023generalization,
  title={Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation},
  author={Zhang, Tianli and Xue, Mengqi and Zhang, Jiangtao and Zhang, Haofei and Wang, Yu and Cheng, Lechao and Song, Jie and Song, Mingli},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={20176--20185},
  year={2023}
}

Contact

Please feel free to contact me via email (zhangtianli@zju.edu.cn) if you are interested in my research :)