Comparisons of different learning rate policy in Caffe

---

Principle

1. fixed: $l{r}_{iter}=l{r}_{base}$

2. step: $l{r}_{iter}=l{r}_{base}\ast {\gamma }^{floor(\frac{iter}{step})}$

3. exp: $l{r}_{iter}=l{r}_{base}\ast {\gamma }^{iter}$

4. inv: $lr_{iter} = lr_{base}(1+\gammaiter)^{-power}$

5. multi_step: $l{r}_{iter}=l{r}_{base}\ast \gamma$, where $\gamma$ is decided by pre-defined steps.

6. poly: $l{r}_{iter}=l{r}_{base}\ast (1-\frac{iter}{maxIter}{)}^{power}$

7. sigmoid: $lr_{iter} = lr_{base}\frac{1}{1+e^{-\gamma(iter-step)}}$

Comparisons