Skip to content

VITA-Group/Orthogonality-in-CNNs

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
October 23, 2018 16:39
November 30, 2018 03:20
December 8, 2018 14:23
October 26, 2018 13:00
January 29, 2020 17:02
October 22, 2018 03:08
August 5, 2019 21:46

Orthogonality in CNNs

Code Implementation for Restricted Isometry Property(RIP) based Orthogonal Regularizers, proposed for Image Classification Task, for various State-of-art ResNet based architectures.

This repositry provides an introduction, implementation and result achieved in the paper: "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?", NeurIPS 2018 [pdf]

Introduction

Orthogonal Network Weights are found to be a favorable property for training deep convolutional neural networks.Through this work, we look to find alternate and more effective ways to enforce orthogonality to deep CNNs. We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences.

Illustration

Can-we-Gain-More-from-Orthogonality Figure 1. Validation Curve Achieved for differnet Regularizers Proposed

Enviroment and Datasets Used

  • Linux
  • Pytorch 4.0
  • Keras 2.2.4
  • CUDA 9.1
  • Cifar10 and Cifar100
  • SVHN
  • ImageNet

Architecture Used

  • ResNet
  • Wide ResNet
  • Pre Resnet
  • ResNext

Regularizers Proposed

  • Single Sided (SO)
  • Double Sided (DSO)
  • Mutual Coherence Based (MC)
  • Restricted Isometry (SRIP) (Best Performing )

Wide-Resnet CIFAR

For CIFAR datasets,we choose Wide Resnet Arch. with a depth of 28 and Kernel width of 10,which gives the best results for comparable number parameters for any Wide-Resnet Model. To train on Cifar-10 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python train_n.py --ngpu 2

To train on Cifar-100 using 2 gpu:

CUDA_VISIBLE_DEVICES=6,7 python train_n.py --ngpu 2 --dataset cifar100

After train phase, you can check saved model in the runs folder.

Wide-Resnet SVHN

For SVHN datasets,we choose Wide Resnet Arch. with a depth of 16 and Kernel width of 8,which gives the best results for comparable number parameters for any Wide-Resnet Model.

CUDA_VISIBEL_DEVICES=0 python train.py --dataset svhn --model wideresnet --learning_rate 0.01 --epochs 160

Result

Network CIFAR-10 CIFAR-100 SVHN
WideResNet 4.16 20.50 1.60
WideResNet + SRIP Reg 3.60 18.19 1.52

Resnet110 CIFAR

We trained CIFAR10 and 100 Dataset for ResNet110 Model and achieved an improvement in terms of Test Accuracy, when compared to a model, which doesn't uses any form Regularization.The Code for this part has been written in Keras, and we have used the base code from official keras Repo: https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py, for a bottleneck based architecture.

Usage

CUDA_VISIBLE_DEVICES=2 python resnet_cifar_new.py

Result

Network CIFAR-10
ResNet110 7.11
WideResNet + SRIP Reg 5.46

Pre-Resnet Imagenet

we trained the Imagenet Dataset for Resnet-34 Resnet 50 and Pre-Resnet 34 and achieved a better Top-5 accuracy when compared to contemporary results. Basic Code was taken from:Pytorch Official cite.

Usage

CUDA_VISIBLE_DEVICES=4,5,6,7 python train_n.py

Result

Network Imagenet Regularizer
PreResnet 34 9.79 NONE
PreResNet 34 8. 85 SRIP
ResNet 34 9.84 NONE
ResNet 34 8.392 SRIP

Pre-Trained Networks

Pre-trained Model are saved on the Google Drive.

Other frameworks

Acknowledgement

References

Citation

If you find our code helpful in your resarch or work, please cite our paper.

@ARTICLE{2018arXiv181009102B,
  author = {{Bansal}, N. and {Chen}, X. and {Wang}, Z.},
   title = "{Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?}",
 journal = {ArXiv e-prints},
archivePrefix = "arXiv",
  eprint = {1810.09102},
keywords = {Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning},
    year = 2018,
   month = oct,
  adsurl = {http://adsabs.harvard.edu/abs/2018arXiv181009102B},
 adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

About

[NeurIPS '18] "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?" Official Implementation.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages