Learning rate decay

[mullachv]

Line of word language modeling 177 should be dividing the learning rate by 4.0 and not 4 (float vs integer) for proper decay of learning rate.

https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L177

[soumith]

which example?

[mullachv]

:) You are too quick for me to even complete writing up my issue:
https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L177

[soumith]

args.lr is of type float. float / int in python is float right?

[mullachv]

But my lr appears to go down to zero, and stay there after epoch 14 (for my particular settings):

| epoch  14 |  2000/ 2323 batches | lr 1.00 | ms/batch 132.84 | loss  3.93 | ppl    51.05
| epoch  14 |  2200/ 2323 batches | lr 1.00 | ms/batch 130.91 | loss  3.87 | ppl    47.75
-----------------------------------------------------------------------------------------
| end of epoch  14 | time: 326.76s | valid loss  4.84 | valid ppl   126.41
-----------------------------------------------------------------------------------------
epoch  15 |   200/ 2323 batches | lr 0.00 | ms/batch 133.49 | loss  4.12 | ppl    61.81
| epoch  15 |   400/ 2323 batches | lr 0.00 | ms/batch 131.63 | loss  4.27 | ppl    71.50
| epoch  15 |   600/ 2323 batches | lr 0.00 | ms/batch 132.10 | loss  4.15 | ppl    63.44
| epoch  15 |   800/ 2323 batches | lr 0.00 | ms/batch 132.22 | loss  4.12 | ppl    61.75
-----------------------------------------------------------------------------------------
| end of epoch  15 | time: 315.69s | valid loss  4.84 | valid ppl   126.41
-----------------------------------------------------------------------------------------
| epoch  16 |   200/ 2323 batches | lr 0.00 | ms/batch 131.21 | loss  4.12 | ppl    61.81
| epoch  16 |   400/ 2323 batches | lr 0.00 | ms/batch 134.99 | loss  4.27 | ppl    71.50

[soumith]

fixed via c4b48c4