Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
img
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Distilling a Powerful Student Model via Online Knowledge Distillation (Link).

The framework of our proposed FFSD for online knowledge distillation. First, student 1 and student 2 learn from each other in a collaborative way. Then by shifting the attention of student 1 and distilling it to student 2, we are able to enhance the diversity among students. Last, the feature fusion module fuses all the students’ information into a fused feature map. The fused representation is then used to assist the learning of the student leader. After training, we simply adopt the student leader which achieves superior performance over all other students.

Getting Started

The code has been tested using Pytorch1.5.1 and CUDA10.2 on Ubuntu 18.04.

Please type the command

pip install -r requirements.txt

to install dependencies.

FFSD

  • You can run the following code to train models on CIFAR-100:

    python cifar.py
    	--dataroot ./database/cifar100
    	--dataset cifar100
    	--model resnet32
    	--lambda_diversity 1e-5
    	--lambda_self_distillation 1000
    	--lambda_fusion 10
    	--gpu_ids 0
    	--name cifar100_resnet32_div1e-5_sd1000_fusion10
  • You can run the following code to train models on ImageNet:

    python distribute_imagenet.py
    	--dataroot ./database/imagenet
    	--dataset imagenet
    	--model resnet18
    	--lambda_diversity 1e-5
    	--lambda_self_distillation 1000
    	--lambda_fusion 10
    	--gpu_ids 0,1
    	--name imagenet_resnet18_div1e-5_sd1000_fusion10

Experimental Results

We provide the student leader models in the experiments, along with their training loggers and configurations.

Model Dataset Top1 Accuracy (%) Download
ResNet20 CIFAR-100 72.64 Link
ResNet20 CIFAR-100 72.58 Link
ResNet20 CIFAR-100 72.88 Link
ResNet32 CIFAR-100 74.92 Link
ResNet32 CIFAR-100 74.82 Link
ResNet32 CIFAR-100 74.82 Link
ResNet56 CIFAR-100 75.84 Link
ResNet56 CIFAR-100 75.66 Link
ResNet56 CIFAR-100 75.91 Link
WRN-16-2 CIFAR-100 75.87 Link
WRN-16-2 CIFAR-100 75.86 Link
WRN-16-2 CIFAR-100 75.69 Link
WRN-40-2 CIFAR-100 79.13 Link
WRN-40-2 CIFAR-100 79.19 Link
WRN-40-2 CIFAR-100 79.11 Link
DenseNet CIFAR-100 77.29 Link
DenseNet CIFAR-100 77.70 Link
DenseNet CIFAR-100 77.17 Link
GoogLeNet CIFAR-100 81.52 Link
GoogLeNet CIFAR-100 81.93 Link
GoogLeNet CIFAR-100 81.34 Link
ResNet-18 ImageNet 70.87 Link
ResNet-34 ImageNet 74.69 Link

You can use the following code to test our models.

python test.py
	--dataroot ./database/cifar100
	--dataset cifar100
	--model resnet32
	--gpu_ids 0
	--load_path ./resnet32/cifar100_resnet32_div1e-5_sd1000_fusion10_1/modelleader_best.pth

Tips

Any problem, free to contact the authors via emails:shaojieli@stu.xmu.edu.cn.

About

Distilling a Powerful Student Model via Online Knowledge Distillation

Resources

Releases

No releases published

Packages

No packages published

Languages