Time Sequence Prediction broken...

[TRTL4LIFE]

When I run the train.py it makes predictions correctly until step 2. Here is the output and graphs generated.

STEP: 0

loss: 1.09054476065
loss: 0.826498360191
loss: 0.742258943851
loss: 0.46692670559
loss: 0.331289575869
loss: 0.236026240101
loss: 0.141259279307
loss: 0.0851531255285
loss: 0.0541043081422
loss: 0.040030847713
loss: 0.0348454304112
loss: 0.0298759609175
loss: 0.0270200336839
loss: 0.0249405874131
loss: 0.0236001364739
loss: 0.0191041066229
loss: 0.0133801897465
loss: 0.011366287259
loss: 0.00983567643843
loss: 0.00893716252079
test loss: 0.00847954020704

STEP: 1

loss: 0.00857080080665
loss: 0.00803037675458
loss: 0.00710759490784
loss: 0.00924047828179
loss: 0.00576260438146
loss: 0.00500472024838
loss: 0.0213943098213
loss: 0.0043065051954
loss: 0.00388057702212
loss: 0.00434915494093
loss: 0.00298646983473
loss: 0.00266532448006
loss: 0.00259056949055
loss: 0.00245869588709
loss: 0.00244078378996
loss: 0.00239072969226
loss: 0.00233532586375
loss: 0.00224324638568
loss: 0.00193656497962
loss: 0.0203101775237
test loss: 0.00190941044782

STEP: 2

loss: 0.0018911771942
loss: 0.00187708194719
loss: 0.00184734283665
loss: 0.00182429561988
loss: 0.0017866811193
loss: 0.00175956313747
loss: 0.0017367104804
loss: 0.0017279887115
loss: 0.00170404503114
loss: 0.00164831568869
loss: 0.0015344472405
loss: 0.00146448434852
loss: 0.00290571241089
loss: 0.00141937563202
loss: 0.00140850693117
loss: 0.00138915262294
loss: 0.00138241696307
loss: 0.00138046501574
loss: 0.00137327773791
loss: 0.00136058323062
test loss: 0.00137266833634

STEP: 3

loss: 0.00133259520575
loss: 0.00126648679697
loss: 0.00142157202624
loss: 0.00132635502659
loss: 0.00136787285234
loss: 0.0011413003146
loss: 0.00110063048445
loss: 0.00103065559065
loss: 0.00102210567153
loss: 0.000941988786438
loss: 0.000857952932802
loss: 0.254636943994
loss: 10.8644616366
loss: 69.6423922962
loss: 66.7292987378
loss: 51.7855249351
loss: 45.1530865189
loss: 349.471334127
loss: 303.620815972
loss: 345.339689524
test loss: 5.94561623697

STEP: 4

loss: 220.910263699
loss: 179.504037927
loss: 162.525490947
loss: 0.500108508365
test loss: 0.502430217481

STEP: 5

loss: 0.500108508365
test loss: 0.502430217481

STEP: 6

loss: 0.500108508365
test loss: 0.502430217481

STEP: 7

loss: 0.500108508365
test loss: 0.502430217481

STEP: 8

loss: 0.500108508365
test loss: 0.502430217481

STEP: 9

loss: 0.500108508365
test loss: 0.502430217481

STEP: 10

loss: 0.500108508365
test loss: 0.502430217481

STEP: 11

loss: 0.500108508365
test loss: 0.502430217481

STEP: 12

loss: 0.500108508365
test loss: 0.502430217481

STEP: 13

loss: 0.500108508365
test loss: 0.502430217481

STEP: 14

loss: 0.500108508365
test loss: 0.502430217481

[sevity]

me too

[boathit]

It seems that this is caused by a too large learning rate in the optimizer. Setting the LBFGS learning rate to a smaller value like 0.8 can address the problem.

[soumith]

could someone send a Pull Request to fix it, if they are confident that it's the right fix.
Thank you very much!

[boathit]

@soumith
Hi Soumith, after changing the learning rate to 0.8, both the training loss and test loss go down steadily and we can reproduce the original result. A Pull Request has been made.

[soumith]

thanks to @boathit , this is now fixed.

[nnmm]

Still happens for me with 0.8. Lower values at least allow the optimizer to function correctly (i. e. perform several steps and monotonically decrease the loss), but the result is still wrong. This is the final plot with lr=0.7:

[barnabytprowe]

I'm also having problems with the example. As is (lr=0.8) my final plot is:
predict14.pdf

and things go wrong from step 1-2:
predict1.pdf

predict2.pdf (looks just like the final plot)

Here's the loss function info on stdout:

STEP:  0
loss: 0.537086230955
loss: 0.519054139134
loss: 0.334312643402
loss: 0.245408088178
loss: 0.241630530524
loss: 0.236310649411
loss: 0.227821229553
loss: 0.212925355116
loss: 0.18457783161
loss: 0.134251495296
loss: 0.0847459995036
loss: 0.0574250318391
loss: 0.0512263487439
loss: 0.0473003375388
loss: 0.039150461162
loss: 0.0352442710805
loss: 0.0323013329392
loss: 0.0247076271594
loss: 0.0190413202173
loss: 0.0149685327246
test loss: 0.0122880477308
STEP:  1
loss: 0.0123111050907
loss: 0.00789734904096
loss: 0.00671599497279
loss: 0.00353805780334
loss: 0.0032202868875
loss: 0.00275785860848
loss: 0.00255092406484
loss: 0.00235055785575
loss: 0.00218358551007
loss: 0.00186726224563
loss: 0.00158670291557
loss: 0.00253018196482
loss: 0.00124947355053
loss: 0.0012916389302
loss: 0.00115403097923
loss: 0.00113527961888
loss: 0.00111034928821
loss: 0.00110101619576
loss: 0.0010856783608
loss: 0.00107512001803
test loss: 0.00151556788033
STEP:  2
loss: 0.00106017296497
loss: 0.00101216329951
loss: 0.195053268076
loss: 1.49997814199
test loss: 1.50362019943
STEP:  3
loss: 1.49997814199
test loss: 1.50362019943
STEP:  4
loss: 1.49997814199
test loss: 1.50362019943
STEP:  5
loss: 1.49997814199
test loss: 1.50362019943
STEP:  6
loss: 1.49997814199
test loss: 1.50362019943
STEP:  7
loss: 1.49997814199
test loss: 1.50362019943
STEP:  8
loss: 1.49997814199
test loss: 1.50362019943
STEP:  9
loss: 1.49997814199
test loss: 1.50362019943
STEP:  10
loss: 1.49997814199
test loss: 1.50362019943
STEP:  11
loss: 1.49997814199
test loss: 1.50362019943
STEP:  12
loss: 1.49997814199
test loss: 1.50362019943
STEP:  13
loss: 1.49997814199
test loss: 1.50362019943
STEP:  14
loss: 1.49997814199
test loss: 1.50362019943

[barnabytprowe]

I'll try some lower values of lr

[barnabytprowe]

For lr=0.1 I again get similar problems as @nnmm found for 0.7 above (but it's an interesting sequence, this is a tricky optimum to find I think): lr_0.1_predicts.tar.gz

[barnabytprowe]

For lr=0.03 things seem to be very slowly moving in the right direction, but 15 steps is insufficient. I'm going to increase the number of steps to 30 and try again: lr_0.03_predicts.tar.gz

[barnabytprowe]

OK so I ended up doing 100 steps / epochs, but it still doesn't converge to a solution like the given example. I can provide the images as fairly low quality jpgs (to come under 10MB) here: lr_0.03_epochs_100_predicts.tar.gz
I also made a movie (gzipped avi): lr_0.03_epochs_100_predicts.avi.gz

It strikes me that this is actually a fairly finely balanced thing to ask the NN to achieve: it is trained to predict the next point only, not a probable arc of points. A lot of decisions can be made to support the first aim that do not necessarily support the second. Thus solutions which achieve attractive long future predictions such as that illustrated in the documentation to this example do not necessarily persist under further training.

Is it asking too much, or am I missing something that ought to make me more optimistic?

[berkersonmez]

Yeah I am getting the exact results as @barnabytprowe.
I wonder how the error goes down with lr=0.1, while it gets seemingly worse results after step 7 according to the pdfs 😕