Implementations of quasi-hyperbolic optimization algorithms.
Clone or download
Latest commit b7f58af Jan 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Quickstart Jan 13, 2019
docs_src Quickstart Jan 13, 2019
qhoptim Initial commit Jan 9, 2019
test_qhoptim Initial commit Jan 9, 2019
.flake8 Initial commit Jan 9, 2019
.gitignore Initial commit Jan 9, 2019
CODE_OF_CONDUCT.rst Initial commit Jan 9, 2019
CONTRIBUTING.rst Initial commit Jan 9, 2019
LICENSE Initial commit Jan 9, 2019
README.rst Quickstart Jan 13, 2019
pyproject.toml Initial commit Jan 9, 2019
setup.py Fix copyright Jan 13, 2019

README.rst

qhoptim: Quasi-hyperbolic optimization

This repository contains PyTorch and TensorFlow implementations of the quasi-hyperbolic momentum (QHM) and quasi-hyperbolic Adam (QHAdam) optimization algorithms from Facebook AI Research.

Quickstart

Use this one-liner for installation:

$ pip install git+https://github.com/facebookresearch/qhoptim.git

Then, you can instantiate the optimizers in PyTorch:

>>> from qhoptim.pyt import QHM, QHAdam

# something like this for QHM
>>> optimizer = QHM(model.parameters(), lr=1.0, nu=0.7, momentum=0.999)

# or something like this for QHAdam
>>> optimizer = QHAdam(
...     model.parameters(), lr=1e-3, nus=(0.7, 1.0), betas=(0.995, 0.999))

Or in TensorFlow:

>>> from qhoptim.tf import QHMOptimizer, QHAdamOptimizer

# something like this for QHM
>>> optimizer = QHMOptimizer(
...     learning_rate=1.0, nu=0.7, momentum=0.999)

# or something like this for QHAdam
>>> optimizer = QHAdamOptimizer(
...     learning_rate=1e-3, nu1=0.7, nu2=1.0, beta1=0.995, beta2=0.999)

Documentation

Please refer to the documentation for installation instructions, usage information, and a Python API reference.

Direct link to installation instructions: here.

Reference

QHM and QHAdam were proposed in the ICLR 2019 paper "Quasi-hyperbolic momentum and Adam for deep learning". We recommend reading the paper for both theoretical insights into and empirical analyses of the algorithms.

If you find the algorithms useful in your research, we ask that you cite the paper as follows:

@inproceedings{ma2019qh,
  title={Quasi-hyperbolic momentum and Adam for deep learning},
  author={Jerry Ma and Denis Yarats},
  booktitle={International Conference on Learning Representations},
  year={2019}
}

Contributing

Bugfixes and contributions are very much appreciated! Please see CONTRIBUTING.rst for more information.

License

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.