HANN

Repository for our preprint VC dimension of partially quantized neural networks in the overparametrized regime.

This repo is ran with the following software (versions)

python 3.8
tensorflow 2.5.0
larq 0.12.0
numpy 1.19.5

Downloading datasets

mkdir results
chmod +x download_UCI_datasets.sh
./download_UCI_datasets.sh

What does the above do?

1. Downloads data_py.zip (39 MB) from 'http://www.bioinf.jku.at/people/klambauer/data_py.zip'. This link is listed at the Self-Normalizing Network repository https://github.com/bioinf-jku/SNNs.
2. Unzips data_py.zip to data/ (268M when unzipped).
3. Cleans up the naming conventions of the subdirectories of data/ and deletes an unused file (data/abalone_dat.py) in the directory.
4. Make a copy of _datasets.py to datasets.py and append the full directory path to the file, so that the data files can be found.

Run experiments

Run everything

Warning! This may take some time. For testing purposes, we recommend running the smaller jobs (see next section). We also have Python/Slurm script for running this on a cluster (see the section after the next section).

python run_batch_experiments.py --datalist all --algorithm HANN15OCE
python run_batch_experiments.py --datalist all --algorithm HANN100OCE

Run a small testing panel

Small testing panel (3 datasets: iris, ionosphere, wine)

python run_batch_experiments.py --datalist testing --algorithm HANN15OCE

Run on a cluster managed by Slurm

The Slurm script is automatically generated by and submitted to your compute cluster by SLURM_run_batch_experiments.py.

You need to manually specify the following variables:

YOUR_EMAIL                    # optional, delete this line if not needed
YOUR_ACCOUNT
COMPLETE_DIRECTORY_TO_HANN    # absolute directory to where this code directory is located on your harddrive

After filling in the above information, run

python SLURM_run_batch_experiments.py --datalist testing --algorithm HANN15OCE --cpus 1

Increase the cpus if you have more cpus per compute node.

Plot figures

The figures are produced in the following jupyter notebooks:

Synthetic data

example_moons.ipynb

example_moons_ste_vs_swish.ipynb

121 UCI datasets

figures_OCE.ipynb

Weston-Watkins

First run

python run_batch_experiments.py --datalist all --algorithm HANN15
python run_batch_experiments.py --datalist all --algorithm HANN100

Then go to figures_WW.ipynb

Other information

wu2018tableA1.csv is Table A1 from the supplemental materials of https://papers.nips.cc/paper/2018/hash/e32c51ad39723ee92b285b362c916ca7-Abstract.html