torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

torchdistill (formerly kdkit) offers various knowledge distillation methods and enables you to design (new) experiments simply by editing a yaml file instead of Python code. Even when you need to extract intermediate representations in teacher/student models, you will NOT need to reimplement the models, that often change the interface of the forward, but instead specify the module path(s) in the yaml file.

Forward hook manager

Using ForwardHookManager, you can extract intermediate representations in model without modifying the interface of its forward function.
This example notebook will give you a better idea of the usage.

Top-1 validation accuracy for ILSVRC 2012 (ImageNet)

T: ResNet-34*	Pretrained	KD	AT	FT	CRD	Tf-KD	SSKD	L2	PAD-L2
S: ResNet-18	69.76*	71.37	70.90	70.45	70.93	70.52	70.09	71.08	71.71
Original work	N/A	N/A	70.70	N/A**	71.17	70.42	71.62	70.90	71.71

* The pretrained ResNet-34 and ResNet-18 are provided by torchvision.
** FT is assessed with ILSVRC 2015 in the original work.
For the 2nd row (S: ResNet-18), the checkpoint (trained weights), configuration and log files are available, and the configurations reuse the hyperparameters such as number of epochs used in the original work except for KD.

Examples

Executable code can be found in examples/ such as

• Image classification: ImageNet (ILSVRC 2012), CIFAR-10, CIFAR-100, etc
• Object detection: COCO 2017, etc
• Semantic segmentation: COCO 2017, etc

Citation

[Preprint]

@article{matsubara2020torchdistill,
  title={torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation},
  author={Matsubara, Yoshitomo},
  year={2020}
  eprint={2011.12913},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}

How to setup

• Python 3.6 >=
• pipenv (optional)

Install by pip/pipenv

pip3 install torchdistill
# or use pipenv
pipenv install torchdistill

Install from this repository

git clone https://github.com/yoshitomo-matsubara/torchdistill.git
cd torchdistill/
pip3 install -e .
# or use pipenv
pipenv install "-e ."

Issues / Contact

The documentation is work-in-progress. In the meantime, feel free to create an issue if you have a feature request or email me ( yoshitom@uci.edu ) if you would like to ask me in private.

References

• 🔍 pytorch/vision/references/classification/
• 🔍 pytorch/vision/references/detection/
• 🔍 pytorch/vision/references/segmentation/
• 🔍 Geoffrey Hinton, Oriol Vinyals and Jeff Dean. "Distilling the Knowledge in a Neural Network" (Deep Learning and Representation Learning Workshop: NeurIPS 2014)
• 🔍 Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta and Yoshua Bengio. "FitNets: Hints for Thin Deep Nets" (ICLR 2015)
• 🔍 Junho Yim, Donggyu Joo, Jihoon Bae and Junmo Kim. "A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning" (CVPR 2017)
• 🔍 Sergey Zagoruyko and Nikos Komodakis. "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer" (ICLR 2017)
• 🔍 Nikolaos Passalis, and Anastasios Tefas. "Learning Deep Representations with Probabilistic Knowledge Transfer" (ECCV 2018)
• 🔍 Jangho Kim, Seonguk Park and Nojun Kwak. "Paraphrasing Complex Network: Network Compression via Factor Transfer" (NeurIPS 2018)
• 🔍 Byeongho Heo, Minsik Lee, Sangdoo Yun and Jin Young Choi. "Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons" (AAAI 2019)
• 🔍 Wonpyo Park, Dongju Kim, Yan Lu and Minsu Cho. "Relational Knowledge Distillation" (CVPR 2019)
• 🔍 Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence and Zhenwen Dai. "Variational Information Distillation for Knowledge Transfer" (CVPR 2019)
• 🔍 Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato and Sameer Singh. "Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems" (Workshop on Hot Topics in Video Analytics and Intelligent Edges: MobiCom 2019)
• 🔍 Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou and Zhaoning Zhang. "Correlation Congruence for Knowledge Distillation" (ICCV 2019)
• 🔍 Frederick Tung and Greg Mori. "Similarity-Preserving Knowledge Distillation" (ICCV 2019)
• 🔍 Yonglong Tian, Dilip Krishnan and Phillip Isola. "Contrastive Representation Distillation" (ICLR 2020)
• 🔍 Yoshitomo Matsubara and Marco Levorato. "Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks" (ICPR 2020)
• 🔍 Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang and Jiashi Feng. "Revisiting Knowledge Distillation via Label Smoothing Regularization" (CVPR 2020)
• 🔍 Guodong Xu, Ziwei Liu, Xiaoxiao Li and Chen Change Loy. "Knowledge Distillation Meets Self-Supervision" (ECCV 2020)
• 🔍 Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang and Yichen Wei. "Prime-Aware Adaptive Distillation" (ECCV 2020)