Add "plateau" LR policy

[sanghoon]

This PR adds a new LR policy called "plateau".
It's believed that as far as loss decreases it's better to keep the higher LR, and this policy help you to do that without repetitive trials or continuous monitoring.
With the policy, LR is lowered when the minimum loss isn't updated for a certain number of iterations (plateau_winsize). On the other hand, the LR never drops if loss keeps decreasing.

You should set one or more window-sizes (just like with "multistep" policy) in solver.prototxt

lr_policy: "plateau"
plateau_winsize: 10000
plateau_winsize: 20000
plateau_winsize: 40000
plateau_winsize: 80000

[sanghoon]

This method has been used in experiments for the following article.
(The exact implementations are slightly different)
PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection

[lukeyeager]

Why not just re-use stepsize and/or stepvalue?

Also, did you mean to use plateau_winsize here? Your PR body mentions plateau_stepsize.

[sanghoon]

Thank you for your comment. I just updated the PR body.

Talking about plateau_winsize, I agree that it's not a bad idea to re-use stepvalue or stepsize. However, they specify exact number of iterations between LR drops while plateau_winsize is more like a minimum bound of of iterations between two consecutive LR drops. I thought sharing one parameter name might cause confusion.

[sanghoon]

Applied changes from the upstream

[JordanPeltier]

Hi,
"It's believed that as far as loss decreases it's better to keep the higher LR" -> Is this loss calculated on training or testing dataset ?
Thanks

[sanghoon]

Hi @JM-MP
The current code supports monitoring 'training loss' only.

[twmht]

@sanghoon

Is this only supported in single gpu mode?

In multi-gpu mode, this->minimum_loss_ may be different in each of gpus.