Demos demonstrations of things that can go wrong when initializing weights
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
plots
.DS_Store
README.md
batchnorm-demo.py
init-demo.py

README.md

nn-init-demo

Shows the "vanishing gradient" problem and "dying ReLU" problems, as well as the problems of the activations going to zero if we do the usual "small, random weight" initialization technique. Then we investigate the Xavier init, He init, and Batch normalization techniques for combatting this issue.