Skip to content
Code for the paper “Characterizing Well-behaved vs. Pathological Deep Neural Networks“ published in 36th International Conference on Machine Learning (ICML 2019): https://arxiv.org/abs/1811.03087
Branch: master
Clone or download
Latest commit 4cd62d4 Jun 19, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
figures Updated markdown in notebooks May 13, 2019
moments_dnns Set 'BN', 'FF', 'Res', 'FC' to lower case May 13, 2019
npy Renamed npy/ folders with lower case May 13, 2019
.gitignore Initial commit Jan 24, 2019
Complements on fully-connected networks.ipynb
Complements on width, boundary conditions, dataset, epsilon.ipynb Updated markdown in notebooks Jun 15, 2019
LICENSE Added LICENSE May 18, 2019
README.md Edited README.md Jun 19, 2019
Reproducing Fig. 2, 3, 4, 5.ipynb Updated markdown in notebooks May 13, 2019
manage_experiments.py Set 'BN', 'FF', 'Res', 'FC' to lower case May 13, 2019
plots.py Set 'BN', 'FF', 'Res', 'FC' to lower case May 13, 2019
requirements.txt Fixed vulnerability May 7, 2019
run_experiment.py Set 'BN', 'FF', 'Res', 'FC' to lower case May 13, 2019

README.md

Moments-DNNs

Code for the paper “Characterizing Well-behaved vs. Pathological Deep Neural Networks” published in 36th International Conference on Machine Learning (ICML 2019): https://arxiv.org/abs/1811.03087

Setup

This package has been tested with python 2.7 and 3.7.

First, you need to install all necessary python dependencies:

cd moments-dnns
sudo pip install -r requirements.txt   # optionally: sudo pip3 install -r requirements.txt

To run reasonably large experiments with convolutional networks, you will need to have tensorflow-gpu installed, e.g. with

sudo pip install tensorflow-gpu

Description

The package is built on top of Keras. At the core of the package, four types of custom layers perform the simultaneous propagation of signal and noise:

  • Convolutional layers
  • Batch normalization layers
  • Activation layers
  • Addition layers to merge residual and skip-connection branches for resnets

There are also custom layers for the computation of the moments of signal and noise. Performing these computations inside the model rather than outside is much more effective both in terms of speed and memory usage.

The entry-point of the package is run_experiment.py. This file contains the function run_experiment() which runs an experiment with fixed parameters for a given number of realizations. The results of the experiment are saved as numpy arrays in the folder npy/name_experiment/ with the parameter name_experiment set at the invocation of run_experiment().

For an experiment with 1,000 realizations .npy files typically occupy a space of a few MB. This space can be optionally reduced by calling the function prune_experiment() in the file manage_experiments.py. This function enables to only retain the moments relevant for a specific type of plot.

The file plots.py provides function to plot the results of the experiments in situations equivalent to Fig. 2, 3, 4, 5 of the paper.

Notebooks

A good way to familiarize with the package is to have a look at the notebooks. The main notebook Reproducing Fig. 2, 3, 4, 5 shows the function calls as well as the results from Fig. 2, 3, 4, 5 of the paper.

There are two complementary notebooks

  • Complements on width, boundary conditions, dataset, epsilon discusses the effect of changing the width, boundary conditions of convolutional layers, input dataset and batch normalization fuzz factor

  • Complements on fully-connected networks discusses the results of experiments equivalent to Fig. 2, 3, 4, 5 for fully-connected networks

These complementary notebooks confirm the results of the paper and provide additional insights and examples of usage of the function run_experiment().

You can’t perform that action at this time.