Skip to content
Reference caffe implementation of LSUV initialization
C++ Python Cuda CMake MATLAB Makefile Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmake Replace unistd functions with cross platform counterparts Nov 10, 2015
data [example] image classification web demo Jul 12, 2014
docs Add parentheses to backward_{cpu,gpu} method. Nov 12, 2015
examples fixed typo Nov 25, 2015
include/caffe Merge pull request #3296 from cdoersch/normalize_batch Nov 23, 2015
matlab Merge pull request #3116 from ronghanghu/solver-refactor Oct 17, 2015
models Switched order of two layers for simpler diff with untuned file Nov 9, 2015
python Fix outs and diffs being overwritten in forward_backward_all. Nov 14, 2015
scripts [style] fix whitespace in travis_install.sh Nov 5, 2015
src Merge pull request #3296 from cdoersch/normalize_batch Nov 23, 2015
tools Make skipping message nicer Jun 2, 2016
.Doxyfile update doxygen config to stop warnings Sep 3, 2014
.gitignore Update example bash scripts to expect .h5, new extensions in .gitignore Aug 7, 2015
.travis.yml Separate IO dependencies Sep 17, 2015
CMakeLists.txt Merge pull request #3221 from BonsaiAI/cmake-forward-compatibility Oct 25, 2015
CONTRIBUTING.md [docs] add CONTRIBUTING.md which will appear on GitHub new Issue/PR p… Jul 30, 2015
CONTRIBUTORS.md clarify the license and copyright terms of the project Aug 7, 2014
INSTALL.md
LICENSE Merge branch 'master' of https://github.com/ducha-aiki/LSUVinit Nov 23, 2015
Makefile Merge pull request #3127 from lukeyeager/install-lib-permissions Nov 11, 2015
Makefile.config.example Merge pull request #3186 from lionleaf/makefile_opencv_imgcodecs Oct 22, 2015
README.md Added link LSUV implementation in thinc library Oct 31, 2017
caffe.cloc [fix] stop cloc complaint about cu type Sep 4, 2014

README.md

Layer-sequential unit-variance (LSUV) initialization

This is sample code for LSUV and Orthonormal initializations, implemented in python script within Caffe framework. The original Caffe README is reproduced below the line. LSUV initialization is described in:

Mishkin, D. and Matas, J.,(2015). All you need is a good init. arXiv preprint arXiv:1511.06422.

Orthonormal initialization is described in:

Andrew M. Saxe, James L. McClelland, Surya Ganguli (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120

upd.: Torch re-implementation https://github.com/yobibyte/torch-lsuv

upd.: Keras re-implementation https://github.com/ducha-aiki/LSUV-keras

New! PyTorch re-implementation https://github.com/ducha-aiki/LSUV-pytorch

New! Thinc re-implementation LSUV-thinc

update: Why it is important to scale your input to var=1 before applying LSUV:

scale-no-scale

Examples

See examples and sample log outputs in examples/lsuv. Run get_datasets.sh and training_*.sh for experiments reproduction. Note, than we haven`t freezed random seed and results may vary a little from run to run. Initialization script itself is in tools/extra/lsuv_init.py

Using with you current Caffe, or different framework installation

If you have Caffe installed already, you can simply copy script from tools/extra/lsuv_init.py and view its help (python lsuv_init.py -help) for further instructions. If you use a different framework, please, adapt that same script to your needs and notify us once you've got the code working, so we could let others know of existance of your solution if you're not against sharing your code. The script is self-explanatory, readable and adaptable.

Citation

Please cite us if you use this code:

@ARTICLE{LSUVInit2015,
author = {{Mishkin}, D. and {Matas}, J.},
title = "{All you need is a good init}",
journal = {arXiv preprint arXiv:1511.06422},
year = 2015,
month = nov
}

Caffe

Build Status License

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and community contributors.

Check out the project site for all the details like

and step-by-step examples.

Join the chat at https://gitter.im/BVLC/caffe

Please join the caffe-users group or gitter chat to ask questions and talk about methods and models. Framework development discussions and thorough bug reports are collected on Issues.

Happy brewing!

License and Citation

Caffe is released under the BSD 2-Clause license. The BVLC reference models are released for unrestricted use.

Please cite Caffe in your publications if it helps your research:

@article{jia2014caffe,
  Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor},
  Journal = {arXiv preprint arXiv:1408.5093},
  Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},
  Year = {2014}
}
You can’t perform that action at this time.