Skip to content
Python
Branch: master
Clone or download
8enmann Merge pull request #6 from ousou/master
Set trust_ratio to 1 if weight_norm or adam_norm is 0.
Latest commit 5ef3ebd Jul 26, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images Match paper's hyperparameters, save a little memory, add clamp, updat… Jun 16, 2019
pytorch_lamb Set trust_ratio to 1 if weight_norm or adam_norm is 0. Jul 26, 2019
.gitignore setup.py and structure changes Apr 28, 2019
.pylintrc initial commit Apr 28, 2019
LICENSE Create LICENSE Apr 28, 2019
README.md Match paper's hyperparameters, save a little memory, add clamp, updat… Jun 16, 2019
setup.py Match paper's hyperparameters, save a little memory, add clamp, updat… Jun 16, 2019
test_lamb.py

README.md

Implementation of https://arxiv.org/abs/1904.00962 for large batch, large learning rate training.

The paper doesn't specify clamp values for ϕ, so I use 10.

Bonus: TensorboardX logging (example below).

Try the sample

git clone git@github.com:cybertronai/pytorch-lamb.git
cd pytorch-lamb
pip install -e .
python test_lamb.py
tensorboard --logdir=runs

Sample results

At --lr=.02, the Adam optimizer is unable to train.

Red: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=adam

Blue: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=lamb

You can’t perform that action at this time.