Implementation of model compression with knowledge distilling method.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
docs Add Hinton's method. Dec 28, 2016
results Add result.(two figures) Jan 3, 2017 Update Jan 3, 2017
student.npy First commit Dec 8, 2016 Correct Hinton method Jan 3, 2017
teacher.npy First commit Dec 8, 2016


Implementation of model compression with three knowledge distilling or teacher student methods [1][2][3].
The basic architecture is teacher-student model.


I used cifar-10 dataset to do this work.

Download cifar-10 dataset



In this the work, I use network in network[5] as teacher model, lenet[6] as student model.
The teacher model is pre-trained by caffe. And extract the model weight by [4].
Both network-in-network and lenet have little different from original model.
In docs, there are two images for the network architecture.

"teacher.npy" is the pre-trained model weights of teacher model.

"student.npy" is the model weights train on lenet, using ground turth label directly.

#Usage In, there is three methods to train student network.
You need to modify the cifar-dataset-path in function read_cifar10

###Basic Usage train by [1]

python --task train --model savemodel

train by [2]

python --task train --model savemodel --noisy [--noisy_ratio --noisy_sigma]

train by [3]

python --task train --model savemodel --KD [--lamda --tau]

**testing** >python --task test --model trained_model
**validation** Also, you can validate your pre-trained teacher model by
> python --task val

This can make sure that your caffe-teacher-model transfer to tensorflow successfully.
python -h for more information


All three methods train 100 epochs, with dropout ratio=0.8, lr=1e-3, decay 0.1 at 80th epoch.
In method[2], noisy_ratio=0.5, sigma=0.1.
In methos[3], lamda=0.3, tau=0.3.

This table shows the accuracy on testing dataset, test by 100-epoch-model.
See more details in result.

method[1] method[2] method[3]
71.97% 70.63% 70.96%

The accuarcy of original model which directly learn by ground truth label:
teacher model : 78.1%
student model : 66.15%


[1] Ba, J. and Caruana, R. Do deep nets really need to be deep? In NIPS 2014.

[2] Bharat Bhusan Sau Vineeth N. Balasubramanian, Deep Model Compression: Distilling Knowledge from Noisy Teachers. arXiv 2016.

[3] Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv 2015.


[5] Network in Network model -

[6] Y. LeCun, L. Bottou, Y. Bengio and P. Haffner: Gradient-Based Learning Applied to Document Recognition, Proceedings of the IEEE 1998