Deep Convolutional Networks as shallow Gaussian Processes
Note for reviewers: there is identifying information in the commit history.
Code for "Deep Convolutional Networks as shallow Gaussian Processes" (OpenReview, by Anonymous Authors.
This package has been tested only with python 3.5 and 3.6.
First, you need to install the package in developer mode. This will download and install all necessary python dependencies:
cd convnets-as-gps # optionally: pip install --user -e . pip install -e .
If you have an "old" CPU, this might crash and return "Illegal Instruction". This is because recent versions of Tensorflow come with AVX instructions enabled. Install tensorflow 1.5.0 to fix this.
Running the experiments
Easy way to run all the experiments: read/run
All the experiments in the paper are run in a two-stage process:
save_kernels_resnet.py, to compute kernel matrices and save them to disk in a working directory. Disk space required: about 15GB for 1 run. Run
python3 program.py --helpfor detailed information, but here are example invocations:
python3 save_kernels.py --seed=<random seed> --n_max=200 --path=/path/to/working/directory python3 save_kernels_resnet.py --n_gpus=1 --n_max=200 --path=/path/to/working/directory
In particular, the
n_max flag determines how many training examples your GPU
processes simultaneously. The memory requirements scale roughly proportionally
n_max^2, adjust the number for your particular hardware.
You might run into "Matrix is singular" errors. In my testing, those can be
removed by reducing
n_max. This must be a bug of some kind in the libraries
that I use (Tensorflow maybe?), but I have no skill or time to acquire the
skill to troubleshoot it. Just reduce
classify_gp.pyto invert the kernel matrix and calculate test results. This requires a lot of CPU RAM memory, at least enough to hold the matrix to invert with 64-bit precision. For MNIST, the main kernel matrix is ~12GB, so you need ~24GB of memory to maintain a decent speed. I'm sure there's a way to do the inverse reasonably fast and more memory-efficiently, but that would take quite a bit of development time.
BibTex citation record
Redacted. Please don't look in the commit history.