No description, website, or topics provided.
Clone or download
Latest commit 26d2fef Jan 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
loss Add files via upload Dec 24, 2018
models/cifar Add files via upload Dec 24, 2018
scripts Add files via upload Dec 24, 2018
utils Update __init__.py Jan 17, 2019
LICENSE Update LICENSE Dec 24, 2018
README.md Update README.md Dec 25, 2018
cifar_baseline.py Add files via upload Dec 24, 2018
cifar_one.py Add files via upload Dec 24, 2018

README.md

Knowledge Distillation by On the fly Native Ensemble (ONE)NeurIPS2018

This is an Pytorch implementation of Xu et al. Knowledge Distillation On the Fly Native Ensemble (ONE) NeurIPS 2018 on Python 2.7, Pytorch 2.0. You may refer to our Vedio and Poster for a quick overview.

ONE

Getting Started

Prerequisites:

  • Datasets: CIFAR100, CIFAR10
  • Python 2.7.
  • Pytorch version == 2.0.0.

Running Experiments

you may need change GPU-ID in scripts, “--gpu-id”, the default is 0.

Training:

For example, to train the ONE model using ResNet-32 or ResNet-110 on CIFAR100, run the the following scripts.

bash scripts/ONE_ResNet32.sh
bash scripts/ONE_ResNet110.sh

To train baseline model using ResNet-32 or ResNet-110 on CIFAR100, run the the following scripts.

bash scripts/Baseline_ResNet32.sh
bash scripts/Baseline_ResNet110.sh

Tip for Stabilizing Model Training

It may help to ramp up [https://arxiv.org/abs/1703.01780] the KL cost in the beginning over the first few epochs until the teacher network starts giving good predictions.

Citation

Please refer to the following if this repository is useful for your research.

Bibtex:

@inproceedings{lan2018knowledge,
  title={Knowledge Distillation by On-the-Fly Native Ensemble},
  author={Lan, Xu and Zhu, Xiatian and Gong, Shaogang},
  booktitle={Advances in Neural Information Processing Systems},
  pages={7527--7537},
  year={2018}
}

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgements

This repository is partially built upon the bearpaw/pytorch-classification repository.