This repository provides the code to conduct the experiments of the paper "Convolutional Spectral Kernel Learning with Generalization Guarantees".
- Python 3.7.4
- Pytorch 1.10.0
- NNI 2.5
- CUDA 10.1.168
- cuDnn 7.6.0
- Gpytorch 1.6.0
- GPU: Nvidia RTX 2080Ti 11G
models.py
implements random features based neural networks, including RFFNet and DSKN, their convoluational versions CRFFNet and CSKN, and their vgg-type versions CRFFNet8 and CSKN8. Meanwhile, models.py implements thetrain()
andtest()
methods for these models.utils.py
implements useful tools for loading primal and resized datasets to feed neural networks.densenet.py
is used to implement deep kernel learning (DKL) from Gpytorch.exp_ablation0_*.py
,exp_ablation1_layers.py
,exp_ablation2_regularizers.py
, andexp_ablation3_convolutional.py
are used to explore the effects of non-stationary spectral kernels, the depth of network, additional regularizers, and convolutional filters on the MNIST dataset, respectively.exp_comparison_CSKN.py
trainsCSKN8
andCRFFNet8
on large image datasets, whileexp_comparison_DKLModel.py
andexp_comparison_PretrainedModels.py
implement and train DKL and popular CNN networks, respectively.ipy_exp_ablation.ipynb
andipy_exp_comparison.ipynb
load and illustrate ablation and comparison experimental results, respectively.tune_model.py
andtune_config.yml
are is used to tune hyperparameters vi NNI.data
folder stores all datasets, includingMNIST, FashionMNIST, CIFAR10, CIFAR100
, andTinyImagenet
.results
folder records experimental results andfigures
folder stores pdf files for illustrating experimental results.
- Run the following scripts for ablation experiments
python3 exp_ablation0_RFFNet.py
python3 exp_ablation0_DSKN.py
python3 exp_ablation1_layers.py --num_layers 1
python3 exp_ablation1_layers.py --num_layers 2
python3 exp_ablation1_layers.py --num_layers 3
python3 exp_ablation1_layers.py --num_layers 4
python3 exp_ablation1_layers.py --num_layers 5
python3 exp_ablation2_regularizers.py
python3 exp_ablation3_convolutional.py --kernel_size 3
python3 exp_ablation3_convolutional.py --kernel_size 5
python3 exp_ablation3_convolutional.py --kernel_size 7
- Run codes in
ipy_exp_ablation.ipynb
to illstrate experimental results.
-
Download
TinyImagenet
dataset intodata
folder and process it bytinyimagenet_preprocess.py
. The other datasetsMNIST, FashionMNIST, CIFAR10, CIFAR100
can be downloaded automatically when they are used for the first time. -
Conduct experiments for
CSKN8
andCRFFNet8
on each dataset
python3 exp_comparison_CSKN.py --model CSKN8 --dataset $dataset_name --epochs 300 --repeates 1
python3 exp_comparison_CSKN.py --model CRFFNet8 --dataset $dataset_name --epochs 300 --repeates 1
- Conduct experiments for
DKL
models on each dataset
python3 exp_comparison_DKLModel.py --dataset $dataset_name --epochs 300 --repeates 1
- Conduct experiments for popular CNN models
resnet, vgg, densenet, shufflenet
on each dataset
python3 exp_comparison_PretrainedModels.py --model resnet --dataset $dataset_name --epochs 300 --repeates 1
python3 exp_comparison_PretrainedModels.py --model vgg --dataset $dataset_name --epochs 300 --repeates 1
python3 exp_comparison_PretrainedModels.py --model densenet --dataset $dataset_name --epochs 300 --repeates 1
python3 exp_comparison_PretrainedModels.py --model shufflenet --dataset $dataset_name --epochs 300 --repeates 1
- Run ipy_exp_comparison.ipynb to demenstrate experimental results.
- Install NNI
pip install nni
- Modify the running script
tune_model.py
and the configuration filetune_config.yml
. - Tune hyparameters vi NNI
nnictl create --config ./tune_config.yml
and visit127.0.0.1:8080
to view resuls.