Search direction

[vdumoulin]

I've completed adding the SearchDirection class to Pylearn2 and I tested it. It seems to work as I intended.

I also added some subclasses (adagrad and momentum, or at least momentum in the way I understand it) to test it a little.

If everything is okay with my pull request, I think there would be some cleanup to do in the SGD class as some of the hacks added over time might be replaced by a SearchDirection subclass.

[vdumoulin]

I've commited some of the changes proposed and commented the others. Thanks for the feedback!

[vdumoulin]

I've commited some of the changes proposed and commented the others. Thanks for the feedback!

[goodfeli]

I think there should not be an LRDecay class. When I said before that you might want to use LRDecay and Momentum at the same time, I didn't mean to hack LRDecay into the momentum class. I meant having learning rate decay implemented as a SearchDirection is a problem because you either have to define an interface for combining two search directions, or you will need to hack learning rate decay into every SearchDirection that you might want to use with it.

The existing interface makes it pretty easy to use arbitrary learning rate schedules. If we merge this PR then we're stuck using one decay rate formula and if we want to make that more flexible, we need to implement the flexibility in ever SearchDirection class that we want to use with learning rate decay.

[goodfeli]

I think there should not be an LRDecay class. When I said before that you might want to use LRDecay and Momentum at the same time, I didn't mean to hack LRDecay into the momentum class. I meant having learning rate decay implemented as a SearchDirection is a problem because you either have to define an interface for combining two search directions, or you will need to hack learning rate decay into every SearchDirection that you might want to use with it.

The existing interface makes it pretty easy to use arbitrary learning rate schedules. If we merge this PR then we're stuck using one decay rate formula and if we want to make that more flexible, we need to implement the flexibility in ever SearchDirection class that we want to use with learning rate decay.

[vdumoulin]

I've made the changes. Concerning the implementation of the momentum class, have you seen my comment?

[vdumoulin]

I've made the changes. Concerning the implementation of the momentum class, have you seen my comment?

[lamblin]

Being able to chain different SearchDirections would make sense to me: we could simply pass the directions dictionary returned by the first one, and use it as the gradients of the next one. This could easily be implemented in a new SearchDirection class. The trickiest part would be chaining the update rules of the different SearchDirections, but I think calling clone with the previous updates dictionary as the replace kwarg would solve that.

[lamblin]

Being able to chain different SearchDirections would make sense to me: we could simply pass the directions dictionary returned by the first one, and use it as the gradients of the next one. This could easily be implemented in a new SearchDirection class. The trickiest part would be chaining the update rules of the different SearchDirections, but I think calling clone with the previous updates dictionary as the replace kwarg would solve that.

[goodfeli]

Which comment about the Momentum class are you referring to? I don't think I've seen it.

Which clone method are you talking about?

Chaining SearchDirections involves some subtleties and I find it a bit hard to think through just off the top of my head.

Under the original interface we discussed, a SearchDirection can safely assume that the optimization algorithm will move in the direction that it returns, and it can assume that the direction it is passed is the gradient. Chaining SearchDirections together breaks both assumptions. Have you thought through what the consequences of that would be for a few different use cases?

[goodfeli]

Which comment about the Momentum class are you referring to? I don't think I've seen it.

Which clone method are you talking about?

Chaining SearchDirections involves some subtleties and I find it a bit hard to think through just off the top of my head.

Under the original interface we discussed, a SearchDirection can safely assume that the optimization algorithm will move in the direction that it returns, and it can assume that the direction it is passed is the gradient. Chaining SearchDirections together breaks both assumptions. Have you thought through what the consequences of that would be for a few different use cases?

[lamblin]

@goodfeli I was trying to define a simple "interface for combining two search directions". Chaining them is my proposal.
Regarding clone, I was thinking about theano.clone.
I do not think that the assumptions you mention are necessary for most of the SearchDirections to work, although I agree that some chaining would not make any sense (I would not use momentum after conjugate gradient, for instance).
In my opinion, if a particular SearchDirection needs the incoming directions to be actual gradients, then it should be first in the chain. If another needs the returned directions to be used directly in the updates, then it should be the last in the chain. If it makes sense to scale differently the directions for different parameters, but only with one scalar scale for each parameter, then the next module should only do that, but I think that decision belongs to the programmer.
Typically, decaying learning rate would be applied last (but maybe not, for instance, after another instance of itself, or another instance of SearchDirection that would have the same purpose). Momentum may be apply after TONGA, and then we can add decaying LR. In fact, I think second-order methods can be safely applied to the gradients returned by TONGA.

[lamblin]

@goodfeli I was trying to define a simple "interface for combining two search directions". Chaining them is my proposal.
Regarding clone, I was thinking about theano.clone.
I do not think that the assumptions you mention are necessary for most of the SearchDirections to work, although I agree that some chaining would not make any sense (I would not use momentum after conjugate gradient, for instance).
In my opinion, if a particular SearchDirection needs the incoming directions to be actual gradients, then it should be first in the chain. If another needs the returned directions to be used directly in the updates, then it should be the last in the chain. If it makes sense to scale differently the directions for different parameters, but only with one scalar scale for each parameter, then the next module should only do that, but I think that decision belongs to the programmer.
Typically, decaying learning rate would be applied last (but maybe not, for instance, after another instance of itself, or another instance of SearchDirection that would have the same purpose). Momentum may be apply after TONGA, and then we can add decaying LR. In fact, I think second-order methods can be safely applied to the gradients returned by TONGA.

[vdumoulin]

@goodfeli

Here's what I wrote about the Momentum subclass:

I must have figured it out wrong. The equations I had were
Theta_t <-- Theta_{t-1} + v_t,
v_t <-- mu_v_{t-1} - epsilon_grad(Theta){t-1}
which needed to be in the form
Theta_t <-- Theta{t-1} - epsilon_search_direction
to be used by SGD, so I factored -epsilon in the expression of v_t, which gave me
v_t <-- - epsilon(grad(Theta_{t-1}) - mu/epsilon_v_{t-1}) = -epsilon * search_direction,
search_direction = grad(Theta_{t-1}) - mu/epsilon*v_{t-1}

[vdumoulin]

@goodfeli

Here's what I wrote about the Momentum subclass:

I must have figured it out wrong. The equations I had were
Theta_t <-- Theta_{t-1} + v_t,
v_t <-- mu_v_{t-1} - epsilon_grad(Theta){t-1}
which needed to be in the form
Theta_t <-- Theta{t-1} - epsilon_search_direction
to be used by SGD, so I factored -epsilon in the expression of v_t, which gave me
v_t <-- - epsilon(grad(Theta_{t-1}) - mu/epsilon_v_{t-1}) = -epsilon * search_direction,
search_direction = grad(Theta_{t-1}) - mu/epsilon*v_{t-1}

[vdumoulin]

(Once I fix the Momentum subclass, would it be better to suppress entirely the momentum option in the SGD class?)

[vdumoulin]

(Once I fix the Momentum subclass, would it be better to suppress entirely the momentum option in the SGD class?)

[goodfeli]

I've gotten fairly busy preparing for the AISTATS deadline next week. Do you mind if I want to finish reviewing this until then?

[goodfeli]

I've gotten fairly busy preparing for the AISTATS deadline next week. Do you mind if I want to finish reviewing this until then?

[vdumoulin]

No problem!

[vdumoulin]

No problem!

[vdumoulin]

I've re-based my commit and finished my implementation of momentum. I've added lots of commentaries to justify the counter-intuitive way used to implement it, so hopefully everything will be clear for those who use it.

I also removed the old way momentum was implemented.

[vdumoulin]

I've re-based my commit and finished my implementation of momentum. I've added lots of commentaries to justify the counter-intuitive way used to implement it, so hopefully everything will be clear for those who use it.

I also removed the old way momentum was implemented.

[goodfeli]

Yes it is. This is what allows you to change the momentum over time, which is very important for getting good results. But it would have to be updated to point to the new momentum coefficient.

[goodfeli]

I'm still not convinced momentum should be a SearchDirection, for the following reasons:

1. The case where learning_rate = 0 is unsupported. learning_rate=0 actually is a useful case to be able to run for debugging. If you notice something bad happening during learning and you aren't sure if it is caused by the learning itself or by one of your callbacks, setting the learning rate to 0 can clear that up. With this implementation you'd need to remember to also disable momentum.
2. Have you tested it to make sure it is numerically stable? Having to take the difference between the gradient and something divided by epsilon at every step is making me nervous. I'm not very good at analyzing numerical stuff but it seems to me like this could cause problems if the momentum coefficient is close to 1, the gradient points in the same direction for several steps, and the learning rate is small.
3. Changing the math makes it harder to understand what we're doing. Users need to understand not just momentum but also the Pylearn2 SearchDirection system and why we had to rearrange things to shove momentum into that system.
4. Since momentum is learning-rate dependent, we're not really computing one search direction anymore. It makes the SearchDirection metaphor inappropriate.
5. Algorithms other than SGD don't use momentum so there is not much to be gained by factoring momentum out of the TrainingAlgorithm.
6. You can't implement Nesterov momentum in this framework, because Nesterov momentum requires taking the gradient at a different point than the current model parameters. So the SGD algorithm is going to need a different mechanism for generalizing momentum anyway.

[goodfeli]

I'm still not convinced momentum should be a SearchDirection, for the following reasons:

1. The case where learning_rate = 0 is unsupported. learning_rate=0 actually is a useful case to be able to run for debugging. If you notice something bad happening during learning and you aren't sure if it is caused by the learning itself or by one of your callbacks, setting the learning rate to 0 can clear that up. With this implementation you'd need to remember to also disable momentum.
2. Have you tested it to make sure it is numerically stable? Having to take the difference between the gradient and something divided by epsilon at every step is making me nervous. I'm not very good at analyzing numerical stuff but it seems to me like this could cause problems if the momentum coefficient is close to 1, the gradient points in the same direction for several steps, and the learning rate is small.
3. Changing the math makes it harder to understand what we're doing. Users need to understand not just momentum but also the Pylearn2 SearchDirection system and why we had to rearrange things to shove momentum into that system.
4. Since momentum is learning-rate dependent, we're not really computing one search direction anymore. It makes the SearchDirection metaphor inappropriate.
5. Algorithms other than SGD don't use momentum so there is not much to be gained by factoring momentum out of the TrainingAlgorithm.
6. You can't implement Nesterov momentum in this framework, because Nesterov momentum requires taking the gradient at a different point than the current model parameters. So the SGD algorithm is going to need a different mechanism for generalizing momentum anyway.

[yoshua]

Nicolas Boulanger may have something to say about point 6. He has
discovered a reformulation
of Nesterov that makes it of the same EXACT form as regular momentum except
for the way
the linear coefficients are chosen.

-- Yoshua

On Wed, Nov 14, 2012 at 11:29 AM, goodfeli notifications@github.com wrote:

I'm still not convinced momentum should be a SearchDirection, for the
following reasons:

The case where learning_rate = 0 is unsupported. learning_rate=0
actually is a useful case to be able to run for debugging. If you notice
something bad happening during learning and you aren't sure if it is caused
by the learning itself or by one of your callbacks, setting the learning
rate to 0 can clear that up. With this implementation you'd need to
remember to also disable momentum.
2.

Have you tested it to make sure it is numerically stable? Having to
take the difference between the gradient and something divided by epsilon
at every step is making me nervous. I'm not very good at analyzing
numerical stuff but it seems to me like this could cause problems if the
momentum coefficient is close to 1, the gradient points in the same
direction for several steps, and the learning rate is small.
3.

Changing the math makes it harder to understand what we're doing.
Users need to understand not just momentum but also the Pylearn2
SearchDirection system and why we had to rearrange things to shove momentum
into that system.
4.

Since momentum is learning-rate dependent, we're not really computing
one search direction anymore. It makes the SearchDirection metaphor
inappropriate.
5.

Algorithms other than SGD don't use momentum so there is not much to
be gained by factoring momentum out of the TrainingAlgorithm.
6.

You can't implement Nesterov momentum in this framework, because
Nesterov momentum requires taking the gradient at a different point than
the current model parameters. So the SGD algorithm is going to need a
different mechanism for generalizing momentum anyway.

—
Reply to this email directly or view it on GitHubhttps://github.com/lisa-lab/pylearn/pull/136#issuecomment-10372718.

[yoshua]

Nicolas Boulanger may have something to say about point 6. He has
discovered a reformulation
of Nesterov that makes it of the same EXACT form as regular momentum except
for the way
the linear coefficients are chosen.

-- Yoshua

On Wed, Nov 14, 2012 at 11:29 AM, goodfeli notifications@github.com wrote:

I'm still not convinced momentum should be a SearchDirection, for the
following reasons:

The case where learning_rate = 0 is unsupported. learning_rate=0
actually is a useful case to be able to run for debugging. If you notice
something bad happening during learning and you aren't sure if it is caused
by the learning itself or by one of your callbacks, setting the learning
rate to 0 can clear that up. With this implementation you'd need to
remember to also disable momentum.
2.

Have you tested it to make sure it is numerically stable? Having to
take the difference between the gradient and something divided by epsilon
at every step is making me nervous. I'm not very good at analyzing
numerical stuff but it seems to me like this could cause problems if the
momentum coefficient is close to 1, the gradient points in the same
direction for several steps, and the learning rate is small.
3.

Changing the math makes it harder to understand what we're doing.
Users need to understand not just momentum but also the Pylearn2
SearchDirection system and why we had to rearrange things to shove momentum
into that system.
4.

Since momentum is learning-rate dependent, we're not really computing
one search direction anymore. It makes the SearchDirection metaphor
inappropriate.
5.

Algorithms other than SGD don't use momentum so there is not much to
be gained by factoring momentum out of the TrainingAlgorithm.
6.

You can't implement Nesterov momentum in this framework, because
Nesterov momentum requires taking the gradient at a different point than
the current model parameters. So the SGD algorithm is going to need a
different mechanism for generalizing momentum anyway.

—
Reply to this email directly or view it on GitHubhttps://github.com/lisa-lab/pylearn/pull/136#issuecomment-10372718.

[dwf]

#2 definitely could pose a problem. #4 I think is a strong argument against momentum-as-search-direction.

[dwf]

#2 definitely could pose a problem. #4 I think is a strong argument against momentum-as-search-direction.

[boulanni]

Yes, I found that the following change of variable simplifies the implementation of Nesterov momentum. It is in the same form as regular momentum in the sense that both velocity and parameter updates depend only on the gradient at the current value of the parameters.

In short:

regular momentum:
(1) v_t = mu * v_t-1 - lr * gradient_f(params_t)
(2) params_t = params_t-1 + v_t
(3) params_t = params_t-1 + mu * v_t-1 - lr * gradient_f(params_t-1)

Nesterov momentum:
(4) v_t = mu * v_t-1 - lr * gradient_f(params_t-1 + mu * v_t-1)
(5) params_t = params_t-1 + v_t

alternate formulation for Nesterov momentum:
(6) v_t = mu * v_t-1 - lr * gradient_f(params_t-1)
(7) params_t = params_t-1 + mu * v_t - lr * gradient_f(params_t-1)
(8) params_t = params_t-1 + mu**2 * v_t-1 - (1+mu) * lr * gradient_f(params_t-1)

So with Theano you can use (1) then either (2) or (7)/(8) to have both options.

[boulanni]

Yes, I found that the following change of variable simplifies the implementation of Nesterov momentum. It is in the same form as regular momentum in the sense that both velocity and parameter updates depend only on the gradient at the current value of the parameters.

In short:

regular momentum:
(1) v_t = mu * v_t-1 - lr * gradient_f(params_t)
(2) params_t = params_t-1 + v_t
(3) params_t = params_t-1 + mu * v_t-1 - lr * gradient_f(params_t-1)

Nesterov momentum:
(4) v_t = mu * v_t-1 - lr * gradient_f(params_t-1 + mu * v_t-1)
(5) params_t = params_t-1 + v_t

alternate formulation for Nesterov momentum:
(6) v_t = mu * v_t-1 - lr * gradient_f(params_t-1)
(7) params_t = params_t-1 + mu * v_t - lr * gradient_f(params_t-1)
(8) params_t = params_t-1 + mu**2 * v_t-1 - (1+mu) * lr * gradient_f(params_t-1)

So with Theano you can use (1) then either (2) or (7)/(8) to have both options.

[goodfeli]

Thanks @boulanni . I will add this to the repo as a note for someone to implement.

[goodfeli]

Thanks @boulanni . I will add this to the repo as a note for someone to implement.

[vdumoulin]

I implemented Nicola's reformulation of Nesterov momentum, hopefully everything is all right!

[vdumoulin]

I implemented Nicola's reformulation of Nesterov momentum, hopefully everything is all right!

[goodfeli]

Looks good. Is there a unit test we should add to make sure this does not break again?

[goodfeli]

Can you restore the init_momentum docstring?

[goodfeli]

I thought we were moving Momentum back into the SGD algorithm.

[goodfeli]

I don't actually see the nesterov_momentum flag used anywhere in the "Files changed" view.

[vdumoulin]

It seems like the last rebase I did messed up BIG TIME: the momentum implementation completely disappeared from the last commit. That would explain the strange disappearance of docstring comments.

Sorry about that, I'll have this fixed soon.

[vdumoulin]

It seems like the last rebase I did messed up BIG TIME: the momentum implementation completely disappeared from the last commit. That would explain the strange disappearance of docstring comments.

Sorry about that, I'll have this fixed soon.

[goodfeli]

ok, @dwf is probably the most knowledgeable person about git in the lab. He can probably help you recover stuff that disappeared. I know how to do a rebase but I don't know how to recover from a rebase gone wrong.

[goodfeli]

ok, @dwf is probably the most knowledgeable person about git in the lab. He can probably help you recover stuff that disappeared. I know how to do a rebase but I don't know how to recover from a rebase gone wrong.

[goodfeli]

@vdumoulin is planning to redo this in several small pull requests rather than one big rebase, so I am closing the PR.

[goodfeli]

@vdumoulin is planning to redo this in several small pull requests rather than one big rebase, so I am closing the PR.