02_training_deep_networks.tex

\section{Training Deep Architectures}

\subsection{The Vanishing/Exploding Gradients Problem}

\subsection{Improved Parameter Initializations}

Previously standard random initialization with mean 0 and standard deviation 1 (citation)

%Vanishing Exploding Gradients problem results in part from poorly chosen initialization values for the network's weights.

Vanishing Exploding Gradients problem results in part from poorly chosen activation functions.

Paper: Bengio, Glorot 2010.

\subsection{Non-saturating Activation Functions}

\subsection{Layer Normalizations}

Batch normalization, batch renormalization, Gradient Clipping, Local response normalization.

\subsection{Pretraining}

\subsection{Faster Optimizers}

\subsection{Regularization}