Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 

Kalman Normalization

By Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang and Liang Lin.

Sun Yat-sen University (SYSU), the Chinese University of Hong Kong (CUHK), SenseTime Group Ltd.

Table of Contents

  1. Results
  2. Introduction
  3. Citation
  4. Dependencies
  5. Usage

Results

  • Under the context of micro-batches(batch size = 2), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); mid line: Group Normalization; bottom line: Kalman Normalization Training curves

  • Under the context of large-batches(batch size = 128), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); bottom line: Kalman Normalization) Training curves

Introduction

This repository contains the original models described in the paper "Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches" (https://arxiv.org/abs/1802.03133). These models are those used in ILSVRC and CIFAR

Citation

If you use these models in your research, please cite:

@article{wang2018batch,
	title={Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches},
	author={Wang, Guangrun and Peng, Jiefeng and Luo, Ping and Wang, Xinjiang and Lin, Liang},
	journal={arXiv preprint arXiv:1802.03133},
	year={2018}
}

Dependencies

  • Python 2.7 or 3
  • TensorFlow >= 1.3.0
  • Tensorpack The code depends on Yuxin Wu's Tensorpack. For convenience, we provide a stable version 'tensorpack-installed' in this repository.
    # install tensorpack locally:
    cd tensorpack-installed
    python setup.py install --user
    

Usage

  • To run Group Normalization, use:
    cd KalmanNorm
    python cifar-gn.py --gpu 0 -n 5 --log_dir gn
    
  • To run Batch Normalization under the context of micro-batches, use:
    cd KalmanNorm
    python cifar-bn-microbatch.py --gpu 0 -n 5 --log_dir  bn-microbatch
    
  • To run Kalman Normalization under the context of micro-batches, use:
    cd KalmanNorm
    python cifar-kn-microbatch.py --gpu 0 -n 5 --log_dir  kn-microbatch
    
  • To run Batch Normalization under the context of large-batches, use:
    cd KalmanNorm
    python cifar-bn-largebatch.py   --gpu 0 -n 18 --log_dir  bn-largebatch
    
  • To run Kalman Normalization under the context of large-batches, use:
    cd KalmanNorm
    python cifar-kn-largebatch.py   --gpu 0 -n 18 --log_dir  kn-largebatch
    
    

About

Code of "Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches"

Resources

Releases

No releases published

Packages

No packages published

Languages