Official Implementation for https://arxiv.org/abs/1810.09102
Switch branches/tags
Nothing to show
Clone or download
Latest commit 8b39e47 Oct 26, 2018
Permalink
Failed to load latest commit information.
FIGS Add files via upload Oct 23, 2018
Imagenet Update train.py Oct 26, 2018
ResNet Add files via upload Oct 26, 2018
ResNext Create train.py Oct 26, 2018
SVHN Delete nn Oct 23, 2018
Wide-Resnet Create temp.py Oct 22, 2018
README.md Update README.md Oct 26, 2018

README.md

Can-we-Gain-More-from-Orthogonality

Code Implementation for Restricted Isometry Property(RIP) based Orthogonal Regularizers, proposed for Image Classification Task, for various State-of-art ResNet based architectures.

This repositry provides an introduction, implementation and result achieved in the paper: "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?", NIPS 2018 [pdf]

Introduction

Orthogonal Network Weights are found to be a favorable property for training deep convolutional neural networks.Through this work, we look to find alternate and more effective ways to enforce orthogonality to deep CNNs. We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences.

Illustration

Can-we-Gain-More-from-Orthogonality Figure 1. Validation Curve Achieved for differnet Regularizers Proposed

Enviroment and Datasets Used

  • Linux
  • Pytorch 4.0
  • Keras 2.2.4
  • CUDA 9.1
  • Cifar10 and Cifar100
  • SVHN
  • ImageNet

Architecture Used

  • ResNet
  • Wide ResNet
  • ResNext

Regularizers Proposed

  • Single Sided (SO)
  • Double Sided (DSO)
  • Mutual Coherence Based (MC)
  • Restricted Isometry (SRIP) (Best Performing )

Wide-Resnet CIFAR

For CIFAR datasets,we choose Wide Resnet Arch. with a depth of 28 and Kernel width of 10,which gives the best results for comparable number parameters for any Wide-Resnet Model. To train on Cifar-10 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python train_n.py --ngpu 2

To train on Cifar-100 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python train_n.py --ngpu 2 --dataset cifar100

After train phase, you can check saved model in the runs folder.

Wide-Resnet SVHN

For SVHN datasets,we choose Wide Resnet Arch. with a depth of 16 and Kernel width of 8,which gives the best results for comparable number parameters for any Wide-Resnet Model.

CUDA_VISIBEL_DEVICES=0 python train.py --dataset svhn --model wideresnet --learning_rate 0.01 --epochs 160

Result

Network CIFAR-10 CIFAR-100 SVHN
WideResNet 4.16 20.50 1.60
WideResNet + SRIP Reg 3.60 18.19 1.52

Resnet110 CIFAR

We trained CIFAR10 and 100 Dataset for ResNet110 Model and achieved an improvement in terms of Test Accuracy, when compared to a model, which doesn't uses any form Regularization.The Code for this part has been written in Keras, and we have used the base code from official keras Repo: https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py, for a bottleneck based architecture.

Usage

CUDA_VISIBLE_DEVICES=2 python resnet_cifar_new.py

Result

Network CIFAR-10
ResNet110 7.11
WideResNet + SRIP Reg 5.46

Pre-Trained Networks

Link will be posted soon!

Other frameworks

Acknowledgement

References

Citation

If you find our code helpful in your resarch or work, please cite our paper.

@ARTICLE{2018arXiv181009102B,
  author = {{Bansal}, N. and {Chen}, X. and {Wang}, Z.},
   title = "{Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?}",
 journal = {ArXiv e-prints},
archivePrefix = "arXiv",
  eprint = {1810.09102},
keywords = {Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning},
    year = 2018,
   month = oct,
  adsurl = {http://adsabs.harvard.edu/abs/2018arXiv181009102B},
 adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}