Decorrelated Batch Normalization
Code for reproducing the results in the following paper:
Decorrelated Batch Normalization
Lei Huang, Dawei Yang, Bo Lang, Jia Deng
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. arXiv:1804.08450
Requirements and Dependency
- Install MAGMA (you can find the instructions in 'Install MAGMA.md' ). Note: MAGMA is required for SVD on GPU. Without MAGMA, you can run the code on CPU only, while all the CNN experiments in the paper are run on GPU.
- Install Torch with CUDA (for GPU). Note that
cutorchshould be compiled with MAGMA support if you have installed MAGMA and set the environments correctly.
- Install cudnn v5.
- Install the dependency
luarocks install optnet
1. Reproduce the results for PCA whitening:
This script will download MNIST automatically and you should put the
./dataset/. The experiment results will be saved at
2. Reproduce the results for MLP architecture:
(1) FIM experiments on YaleB dataset
- Prepare the data: download the YaleB dataset here, and put the data files under
/dataset/so that the paths look like
The experiment results will be saved at directory: 'set_result/MLP/'.
You can experiment with different hyperparameters by running these scripts --
(2) Experiments on PIE dataset
- Prepare the data: download the PIE dataset here, and put the data file under
./dataset/such that the paths look like
- To experiment with different group sizes, run:
- To obtain different baseline performances, execute:
bash execute_MLP_2PIE.sh bash execute_MLP_2PIE_nnn.sh
Note that the experiments until this point can be run on CPU, so MAGMA is not needed in above experiments.
3. Reproduce the results for VGG-A architecture on CIFAR-10:
- Prepare the data: follow the instructions for CIFAR-10 in this project . It will generate a preprocessed dataset and save a 1400MB file. Put this file
bash execute_Conv_1vggA_2test_adam.sh bash execute_Conv_1vggA_2test_base.sh bash execute_Conv_1vggA_2test_ELU.sh bash execute_Conv_1vggA_2test_var.sh
Note that if your machine has fewer than 4 GPUs, the environment variable
CUDA_VISIBLE_DEVICES should be changed accordingly.
4. Analyze the properties of DBN on CIFAR-10 datset:
- Prepare the data: same as in VGG-A experiments.
bash exp_Conv_4Splain_1deep.lua bash exp_Conv_4Splain_2large.lua
5. Reproduce the ResNet experiments on CIFAR-10 datset:
bash execute_Conv_2residual_old.sh bash execute_Conv_3residual_wide_Cifar100_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh bash execute_Conv_3residual_wide_Cifar100_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh bash execute_Conv_3residual_wide_Cifar10_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh bash execute_Conv_3residual_wide_Cifar10_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh
6. Reproduce the ImageNet experiments.
- Clone Facebook's ResNet repo here.
- Download ImageNet and put it in:
/tmp/dataset/ImageNet/(you can also customize the path in
- Install the DBN module to Torch as a Lua package: go to the directory
luarocks make cudbn-1.0-0.rockspec.
- Copy the model definitions in
./modelsdirectory in the cloned repo
fb.resnet.torch, for reproducing the results reported in the paper. You also can compare the pre-activation version of residual networks introduced in the paper (using the model files
- Use the default configuration and our models to run experiments.
Email: email@example.com. Any discussions and suggestions are welcome!