Skip to content
Code of "Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches"
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
KalmanNorm
results
tensorpack-installed
.gitignore
README.md

README.md

Kalman Normalization

By Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang and Liang Lin.

Sun Yat-sen University (SYSU), the Chinese University of Hong Kong (CUHK), SenseTime Group Ltd.

Table of Contents

  1. Results
  2. Introduction
  3. Citation
  4. Dependencies
  5. Usage

Results

  • Under the context of micro-batches(batch size = 2), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); mid line: Group Normalization; bottom line: Kalman Normalization Training curves

  • Under the context of large-batches(batch size = 128), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); bottom line: Kalman Normalization) Training curves

Introduction

This repository contains the original models described in the paper "Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches" (https://arxiv.org/abs/1802.03133). These models are those used in ILSVRC and CIFAR

Citation

If you use these models in your research, please cite:

@article{wang2018batch,
	title={Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches},
	author={Wang, Guangrun and Peng, Jiefeng and Luo, Ping and Wang, Xinjiang and Lin, Liang},
	journal={arXiv preprint arXiv:1802.03133},
	year={2018}
}

Dependencies

  • Python 2.7 or 3
  • TensorFlow >= 1.3.0
  • Tensorpack The code depends on Yuxin Wu's Tensorpack. For convenience, we provide a stable version 'tensorpack-installed' in this repository.
    # install tensorpack locally:
    cd tensorpack-installed
    python setup.py install --user
    

Usage

  • To run Group Normalization, use:
    cd KalmanNorm
    python cifar-gn.py --gpu 0 -n 5 --log_dir gn
    
  • To run Batch Normalization under the context of micro-batches, use:
    cd KalmanNorm
    python cifar-bn-microbatch.py --gpu 0 -n 5 --log_dir  bn-microbatch
    
  • To run Kalman Normalization under the context of micro-batches, use:
    cd KalmanNorm
    python cifar-kn-microbatch.py --gpu 0 -n 5 --log_dir  kn-microbatch
    
  • To run Batch Normalization under the context of large-batches, use:
    cd KalmanNorm
    python cifar-bn-largebatch.py   --gpu 0 -n 18 --log_dir  bn-largebatch
    
  • To run Kalman Normalization under the context of large-batches, use:
    cd KalmanNorm
    python cifar-kn-largebatch.py   --gpu 0 -n 18 --log_dir  kn-largebatch
    
    
You can’t perform that action at this time.