Neural Network Pruning

Lucas Liebenwein, Cenk Baykal, Igor Gilitschenski, Harry Lang, Dan Feldman, Daniela Rus

Papers

This repository contains code to reproduce the results from the following papers:

Paper	Venue	Title & Link
Lost	MLSys 2021	Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
PFP	ICLR 2020	Provable Filter Pruning for Efficient Neural Networks
SiPP	arXiv	SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks

Packages

In addition, the repo also contains two stand-alone python packages that can be used for any desired pruning experiment:

Packages	Location	Description
torchprune	./src/torchprune	This package can be used to run any of the implemented pruning algorithms. It also contains utilities to use pre-defined networks (or use your own network) and utilities for standard datasets.
experiment	./src/experiment	This package can be used to run pruning experiments and compare multiple pruning methods for different prune ratios. Each experiment is configured using a .yaml-configuration files.

Paper Reproducibility

The code for each paper is implemented in the respective packages. In addition, for each paper we have a separate folder that contains additional information about the paper and scripts and parameter configuration to reproduce the exact results from the paper.

Paper	Location
Lost	paper/lost
PFP	paper/pfp
SiPP	paper/sipp

Setup

We provide three ways to install the codebase:

1. Github repo + full conda environment
2. Installation via pip
3. Docker image

1. Github Repo

Clone the github repo:

git pull git@github.com:lucaslie/torchprune.git
# (or your favorite way to pull a repo)

We recommend installing the packages in a separate conda environment. Then to create a new conda environment run

conda create -n prune python=3.8 pip
conda activate prune

To install all required dependencies and both packages, run:

pip install -r misc/requirements.txt

Note that this will also install pre-commit hooks for clean commits :-)

2. Pip Installation

To separately install each package with minimal dependencies without cloning the repo manually, run the following commands:

# "torchprune" package
pip install git+https://github.com/lucaslie/torchprune/#subdirectory=src/torchprune

# "experiment" package
pip install git+https://github.com/lucaslie/torchprune/#subdirectory=src/experiment

Note that the experiment package does not automatically install the torchprune package.

3. Docker Image

You can simply pull the docker image from our docker hub:

docker pull liebenwein/torchprune

You can run it interactively with

docker run -it liebenwein/torchprune bash

For your reference you can find the Dockerfile here.

More Information and Usage

Check out the following READMEs in the sub-directories to find out more about using the codebase.

READMEs	More Information
src/torchprune/README.md	more details to prune neural networks, how to use and setup the data sets, how to implement custom pruning methods, and how to add your data sets and networks.
src/experiment/README.md	more details on how to configure and run your own experiments, and more information on how to re-produce the results.
paper/lost/README.md	check out for more information on the Lost paper.
paper/pfp/README.md	check out for more information on the PFP paper.
paper/sipp/README.md	check out for more information on the SiPP paper.

Citations

Please cite the respective papers when using our work.

Lost In Pruning (Pruning Study)

@article{liebenwein2021lost,
title={Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy},
author={Liebenwein, Lucas and Baykal, Cenk and Carter, Brandon and Gifford, David and Rus, Daniela},
journal={Proceedings of Machine Learning and Systems},
volume={3},
year={2021}
}

Provable Filter Pruning (Filter Pruning)

@inproceedings{liebenwein2020provable,
title={Provable Filter Pruning for Efficient Neural Networks},
author={Lucas Liebenwein and Cenk Baykal and Harry Lang and Dan Feldman and Daniela Rus},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=BJxkOlSYDH}
}

SiPPing Neural Networks (Weight Pruning)

@article{baykal2019sipping,
title={SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks},
author={Baykal, Cenk and Liebenwein, Lucas and Gilitschenski, Igor and Feldman, Dan and Rus, Daniela},
journal={arXiv preprint arXiv:1910.05422},
year={2019}
}