Skip to content
SGD with compressed gradients and error-feedback: https://arxiv.org/abs/1901.09847
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
models
notebooks
optimizers
utils
LICENSE
README.md
__init__.py
main.py
main_experiments.py
norm_ratio_experiments.py
requirements.txt
tune_lr.py

README.md

Error-feedback SGD

We present here the code of the experimental parts of the paper Error Feedback Fixes SignSGD and other Gradient Compression Schemes.

The implementation is based on this repository's code and uses PyTorch.

Requirements

The following packages were used for the experiments. Newer versions are also likely to work.

  • torchvision==0.2.1
  • numpy==1.15.4
  • torch==0.4.1
  • pandas==0.23.4
  • scikit_learn==0.20.3

To install them automatically: pip install -r requirements.txt

Organization

  • notebooks/ contains jupyter notebook files with plotted results and experiments.
  • optimizers/ contains the custom optimizer, namely ErrorFeedbackSGD.
  • models/ contains the deep net architectures. Only VGG and Resnet were experimented.
  • results/ contains the results of the experiments in pickle files.
  • utils/ contains utility functions for saving/loading objects, convex optimization, progress bar...
  • checkpoints/ contains the saved models' checkpoints with all the nets parameters. The folder is empty here as those files are very large.

Notations

A few notations in the code don't match the notations from the paper. In particular,

  • What is called signSGD in the paper is the scaled sign SGD in the code, as the gradients are rescaled by their norm.
  • What is called ef-signSGD in the paper is scaled sign SGD with memory in the code. The memory parameter can also be used with other compressions besides the sign.

Usage

  • main.py can be called from the command line to run a single network training and testing. It can take a variety of optional arguments. Type python main.py --help for further details.
  • utils.hyperparameters.py facilitate the definition of all the hyper-parameters of the experiments.
  • tune_lr.py allows to tune the learning rate for a network architecture/data set/optimizer configuration.
  • main_experiments.py contains the experiments presented in the paper, section 6.
You can’t perform that action at this time.