start decay after resume without "-continue"

[vince62s]

Here is the scenario.
first training:
5 epochs, lr 1.0, start decay 5 decay_rate 0.5

Then resume training with -"continue" works fine.
start_epoch 6 end_epoch 9 ==> lr of epoch 6 = 0.5

if I don't use "-continue"
lr of epoch 6 will start at 1 and not decay right away eventhough I am passing options.
I could always use -continue, but I guess one might want to change the decay rate after a resume.

clear ?

[guillaumekln]

The learning rate is only updated at the end of an epoch. So it will complete epoch 6 first, then a first decay will be applied.

[vince62s]

well if I specify start_epoch 6 end_epoch 9 start_decay 5 decay_rate 0.5
without continue it should decay right away, no ?
Otherwise we have to manually adapt the initial learning rate.

[guillaumekln]

Why are you not using -continue in this case?

I believe the current behavior is correct. For example if -start_decay 2 -start_epoch 6 are set, do you expect the code to replay to entire decay history?

[vince62s]

then to avoid confusion, if continue is not set and if you load from an existing model, I f would suggest we throw an error if start_decay_at if less or equal to start_epoch.
if you reset the lr and decay, I don't exactly undertstand the possibility to start from epoch X when load an existing model. do you see my point?

[guillaumekln]

Unless I'm missing something, that is reason we introduced the -continue option, to continue exactly where a checkpoint left off.

When you don't use -continue, it is actually a new training which uses the parameters from a checkpoint independently from their optimization history. It has other use cases like if you want to change the data and set a higher learning rate or change the optimization method.

[vince62s]

ok best to discuss this on the forum.