Official Implementation for
Switch branches/tags
Nothing to show
Clone or download
Latest commit 8b39e47 Oct 26, 2018
Failed to load latest commit information.
FIGS Add files via upload Oct 23, 2018
Imagenet Update Oct 26, 2018
ResNet Add files via upload Oct 26, 2018
ResNext Create Oct 26, 2018
SVHN Delete nn Oct 23, 2018
Wide-Resnet Create Oct 22, 2018 Update Oct 26, 2018


Code Implementation for Restricted Isometry Property(RIP) based Orthogonal Regularizers, proposed for Image Classification Task, for various State-of-art ResNet based architectures.

This repositry provides an introduction, implementation and result achieved in the paper: "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?", NIPS 2018 [pdf]


Orthogonal Network Weights are found to be a favorable property for training deep convolutional neural networks.Through this work, we look to find alternate and more effective ways to enforce orthogonality to deep CNNs. We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences.


Can-we-Gain-More-from-Orthogonality Figure 1. Validation Curve Achieved for differnet Regularizers Proposed

Enviroment and Datasets Used

  • Linux
  • Pytorch 4.0
  • Keras 2.2.4
  • CUDA 9.1
  • Cifar10 and Cifar100
  • SVHN
  • ImageNet

Architecture Used

  • ResNet
  • Wide ResNet
  • ResNext

Regularizers Proposed

  • Single Sided (SO)
  • Double Sided (DSO)
  • Mutual Coherence Based (MC)
  • Restricted Isometry (SRIP) (Best Performing )

Wide-Resnet CIFAR

For CIFAR datasets,we choose Wide Resnet Arch. with a depth of 28 and Kernel width of 10,which gives the best results for comparable number parameters for any Wide-Resnet Model. To train on Cifar-10 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python --ngpu 2

To train on Cifar-100 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python --ngpu 2 --dataset cifar100

After train phase, you can check saved model in the runs folder.

Wide-Resnet SVHN

For SVHN datasets,we choose Wide Resnet Arch. with a depth of 16 and Kernel width of 8,which gives the best results for comparable number parameters for any Wide-Resnet Model.

CUDA_VISIBEL_DEVICES=0 python --dataset svhn --model wideresnet --learning_rate 0.01 --epochs 160


Network CIFAR-10 CIFAR-100 SVHN
WideResNet 4.16 20.50 1.60
WideResNet + SRIP Reg 3.60 18.19 1.52

Resnet110 CIFAR

We trained CIFAR10 and 100 Dataset for ResNet110 Model and achieved an improvement in terms of Test Accuracy, when compared to a model, which doesn't uses any form Regularization.The Code for this part has been written in Keras, and we have used the base code from official keras Repo:, for a bottleneck based architecture.




Network CIFAR-10
ResNet110 7.11
WideResNet + SRIP Reg 5.46

Pre-Trained Networks

Link will be posted soon!

Other frameworks




If you find our code helpful in your resarch or work, please cite our paper.

  author = {{Bansal}, N. and {Chen}, X. and {Wang}, Z.},
   title = "{Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?}",
 journal = {ArXiv e-prints},
archivePrefix = "arXiv",
  eprint = {1810.09102},
keywords = {Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning},
    year = 2018,
   month = oct,
  adsurl = {},
 adsnote = {Provided by the SAO/NASA Astrophysics Data System}