sps-pytorch

Please find the official implementation of the authors here.

This repo aims to reproduce and extend the SPS optimizer by Nicolas Loizou, Sharan Vaswani, Issam Laradji and Simon Lacoste-Julien presented in their paper titled Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence.

Results

RNN-T

Run

• clone and patch fastai

# clone
git clone https://github.com/fastai/fastai
cd fastai
git checkout 1.0.60
pip install -e ".[dev]"

# patch
git apply fastai.patch

• python train.py --opt sps --epochs 5

Resources

• official Implementation of SPS

• mgrankin/over9000 (code snippets)