Find Best Learning Rate On Plateau
A Keras callback to automatically adjust the learning rate when it stops improving.
This method works on the principal that any particular learning rate is only good until it stops improving your loss. At that point you need to select a new learning rate, typically this is done by a decay function (as in
keras.callbacks.ReduceLROnPlateau). The method used here is to simply search for the best learning rate by training for a few epochs, increasing the learning rate as we go and selecting the one that gives the largest loss improvement. The process is something like this:
- Train a model for a large number of epochs
- If the model's loss fails to improve for
- Take a snapshot of the model
- Set training rate to
min_lrand train for a batch
- Increase the learning rate exponentially toward
max_lrafter every batch.
- Once candidate learning rates have been exhausted, select
new_lras the learning rate that gave the steepest negative gradient in loss.
- Reload weights from the snapshot
- Set model's learning rate to
new_lrand continue training as normal
I initially tried performing this process every
n epochs, but it later occurred to me that I should do it only when loss stops improving (a la ReduceLROnPlateau), which is what this repository does.