Code for Decorrelated Batch Normalization
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
dataset add the dir Mar 11, 2018
models add the original residual network experiments (50 and 101 layers) Mar 25, 2018
module initial commit Mar 11, 2018
set_result add the dir Mar 11, 2018
Install_MAGMA.md Update Install_MAGMA.md Apr 24, 2018
LICENSE Initial commit Mar 11, 2018
README.md Update README.md May 12, 2018
augmentation.lua initial commit Mar 11, 2018
dataset-mnist.lua initial commit Mar 11, 2018
execute_Conv_1vggA_1validate_adam.sh initial commit Mar 11, 2018
execute_Conv_1vggA_1validate_sgd.sh initial commit Mar 11, 2018
execute_Conv_1vggA_2test_ELU.sh initial commit Mar 11, 2018
execute_Conv_1vggA_2test_adam.sh initial commit Mar 11, 2018
execute_Conv_1vggA_2test_base.sh initial commit Mar 11, 2018
execute_Conv_1vggA_2test_var.sh initial commit Mar 11, 2018
execute_Conv_2residual_old.sh initial commit Mar 11, 2018
execute_Conv_3residual_wide_Cifar100_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh initial commit Mar 11, 2018
execute_Conv_3residual_wide_Cifar100_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh initial commit Mar 11, 2018
execute_Conv_3residual_wide_Cifar10_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh initial commit Mar 11, 2018
execute_Conv_3residual_wide_Cifar10_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh initial commit Mar 11, 2018
execute_Conv_4Splain_1deep.sh initial commit Mar 11, 2018
execute_Conv_4Splain_2large.sh initial commit Mar 11, 2018
execute_MLP_0debug_MNIST.sh initial commit Mar 11, 2018
execute_MLP_1FIM_YaleB_HyperP.sh initial commit Mar 11, 2018
execute_MLP_1FIM_YaleB_HyperP_nnn.sh initial commit Mar 11, 2018
execute_MLP_1FIM_YaleB_best.sh initial commit Mar 11, 2018
execute_MLP_2PIE.sh initial commit Mar 11, 2018
execute_MLP_2PIE_DBNGroup.sh initial commit Mar 11, 2018
execute_MLP_2PIE_nnn.sh initial commit Mar 11, 2018
exp_Conv_1vggA_1validate.lua initial commit Mar 11, 2018
exp_Conv_1vggA_2test.lua initial commit Mar 11, 2018
exp_Conv_2residual_old.lua initial commit Mar 11, 2018
exp_Conv_3residual_wide_Cifar10.lua initial commit Mar 11, 2018
exp_Conv_3residual_wide_Cifar100.lua initial commit Mar 11, 2018
exp_Conv_4Splain_1deep.lua initial commit Mar 11, 2018
exp_Conv_4Splain_2large.lua initial commit Mar 11, 2018
exp_MLP_0debug_MNIST.lua initial commit Mar 11, 2018
exp_MLP_1FIM_YaleB.lua initial commit Mar 11, 2018
exp_MLP_2perform_PIE.lua initial commit Mar 11, 2018
provider.lua initial commit Mar 11, 2018

README.md

Decorrelated Batch Normalization

Code for reproducing the results in the following paper:

Decorrelated Batch Normalization
Lei Huang, Dawei Yang, Bo Lang, Jia Deng
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. arXiv:1804.08450

Requirements and Dependency

  • Install MAGMA (you can find the instructions in  'Install MAGMA.md' ). Note: MAGMA is required for SVD on GPU. Without MAGMA, you can run the code on CPU only, while all the CNN experiments in the paper are run on GPU.
  • Install Torch with CUDA (for GPU). Note that cutorch should be compiled with MAGMA support if you have installed MAGMA and set the environments correctly.
  • Install cudnn v5.
  • Install the dependency optnet by:
luarocks install optnet

Experiments

1. Reproduce the results for PCA whitening:

  • Run:
bash execute_MLP_0debug_MNIST.sh

This script will download MNIST automatically and you should put the mnist.t7/ under ./dataset/. The experiment results will be saved at ./set_result/MLP/.

2. Reproduce the results for MLP architecture:

(1) FIM experiments on YaleB dataset
  • Prepare the data: download the YaleB dataset here, and put the data files under /dataset/ so that the paths look like ./dataset/YaleB/YaleB_train.dat and ./dataset/YaleB/YaleB_test.dat.
  • Run:
bash execute_MLP_1FIM_YaleB_best.sh

The experiment results will be saved at directory: 'set_result/MLP/'.

You can experiment with different hyperparameters by running these scripts -- execute_MLP_1FIM_YaleB_HyperP.sh and execute_MLP_1FIM_YaleB_HyperP_nnn.sh.

(2) Experiments on PIE dataset
  • Prepare the data: download the PIE dataset here, and put the data file under ./dataset/ such that the paths look like ./dataset/PIE/PIE_train.dat and ./dataset/PIE/PIE_test.dat.
  • To experiment with different group sizes, run:
bash execute_MLP_2PIE_DBNGroup.sh
  • To obtain different baseline performances, execute:
 bash execute_MLP_2PIE.sh
 bash execute_MLP_2PIE_nnn.sh

Note that the experiments until this point can be run on CPU, so MAGMA is not needed in above experiments.


3. Reproduce the results for VGG-A architecture on CIFAR-10:

  • Prepare the data: follow the instructions for CIFAR-10 in this project . It will generate a preprocessed dataset and save a 1400MB file. Put this file cifar_provider.t7 under ./dataset/.
  • Run:
bash execute_Conv_1vggA_2test_adam.sh
bash execute_Conv_1vggA_2test_base.sh
bash execute_Conv_1vggA_2test_ELU.sh
bash execute_Conv_1vggA_2test_var.sh

Note that if your machine has fewer than 4 GPUs, the environment variable CUDA_VISIBLE_DEVICES should be changed accordingly.

4. Analyze the properties of DBN on CIFAR-10 datset:

  • Prepare the data: same as in VGG-A experiments.
  • Run:
bash exp_Conv_4Splain_1deep.lua
bash exp_Conv_4Splain_2large.lua

5. Reproduce the ResNet experiments on CIFAR-10 datset:

  • Prepare the data: download CIFAR-10 and CIFAR-100, and put the data files under ./dataset/.
  • Run:
bash execute_Conv_2residual_old.sh
bash execute_Conv_3residual_wide_Cifar100_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh
bash execute_Conv_3residual_wide_Cifar100_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh
bash execute_Conv_3residual_wide_Cifar10_wr_BN_d28_h48_g16_b128_dr0.3_s1_C2.sh
bash execute_Conv_3residual_wide_Cifar10_wr_DBN_scale_L1_d28_h48_g16_b128_dr0.3_s1_C3.sh

6. Reproduce the ImageNet experiments.

  • Clone Facebook's ResNet repo here.
  • Download ImageNet and put it in: /tmp/dataset/ImageNet/ (you can also customize the path in opts.lua)
  • Install the DBN module to Torch as a Lua package: go to the directory ./models/imagenet/cuSpatialDBN/ and run luarocks make cudbn-1.0-0.rockspec.
  • Copy the model definitions in ./models/imagenet/ (resnet_BN.lua, resnet_DBN_scale_L1.lua and init.lua) to ./models directory in the cloned repo fb.resnet.torch, for reproducing the results reported in the paper. You also can compare the pre-activation version of residual networks introduced in the paper (using the model files preresnet_BN.lua and preresnet_DBN_scale_L1.lua).
  • Use the default configuration and our models to run experiments.

Contact

Email: huanglei@nlsde.buaa.edu.cn. Any discussions and suggestions are welcome!