nesterov momentum is wrong in sgd

[soumith]

@clementfarabet elaborate here?

[clementfarabet]

Yeah I think the nesterov mode in optim.sgd is missing a step. Here's a self-contained version of Nesterov Accelerated Gradient that I think is correct (I used it for a while, @dilipkay wrote it a while back).

But maybe I'm misreading the nesterov option in optim.sgd. @fidlej , it looks like you added this parameter. Was the intent to implement the NAG method or something slightly different? Thanks!

[fidlej]

If the learning rate is a constant, the implemented nesterov mode should behave like NAG.
The used update rule is derived here.
The state.dfdx in the code is the same as v_{t+1}/learningRate. I.e., state.dfdx is a decaying sum of the gradients.

[koraykv]

I suggest we keep this one as it is. I think it is not wrong, but an approximation. The gist @clementfarabet posted is another implementation which I will put in a separate file.

[clementfarabet]

Thanks for the clarification Ivo.

Once Koray adds it in we should probably compare them.

Clément

On Oct 13, 2014, at 1:00 PM, koray kavukcuoglu notifications@github.com wrote:

Closed #27.

—
Reply to this email directly or view it on GitHub.

[koraykv]

I just added it in.

[clementfarabet]

Cool, thanks Koray!