LSTMTimeStep can't forecast a simple array of numbers...

[philippscheer]

What is wrong?

LSTMTimeStep cannot correctly predict a simple array of numbers

Where does it happen?

When I want to use forecast()

How do we replicate the issue?

Just use my code, it's simple:

	const net = new brain.recurrent.LSTMTimeStep();
	const trainingConfig = {
								// Defaults values --> expected validation
		  iterations: 10000,    // the maximum times to iterate the training data --> number greater than 0
		  errorThresh: 0.00000000001,   // the acceptable error percentage from training data --> number between 0 and 1
		  log: false,           // true to use console.log, when a function is supplied it is used --> Either true or a function
		  logPeriod: 100,        // iterations between logging out --> number greater than 0
		  learningRate: 0.3,    // scales with delta to effect training rate --> number between 0 and 1
		  momentum: 0.1,        // scales with next layer's change value --> number between 0 and 1
		  callback: null,       // a periodic call back that can be triggered while training --> null or function
		  callbackPeriod: 10,   // the number of iterations through the training data between callback calls --> number greater than 0
		  timeout: Infinity     // the max number of milliseconds to train for --> number greater than 0
	}
	const training = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
	net.train([
		training
	], trainingConfig);
	const output = net.forecast(training)
	console.log(output);

How important is this (1-5)?

2

Expected behavior (i.e. solution)

It should output 11... (+-0.5)

Other Comments

When I use forecast(input), it gives me different results based on the iteration count. That's what I noticed (first number is iterations second one is the output => should be 11):
500: 13.98
1000: 9.5
2000: 12.87
3000: 10.74
10000: 9.84

[mubaidr]

I believe this is the issue with the learning rate value at default value this runs as expected:
http://jsfiddle.net/mubaidr/8yeo2jmd/10/

This is some strange behavior, output change dramatically between different runs:
http://jsfiddle.net/mubaidr/8yeo2jmd/21/

@robertleeplummerjr can confirm please?

[robertleeplummerjr]

This is some strange behavior, output change dramatically between different runs:
http://jsfiddle.net/mubaidr/8yeo2jmd/21/

@mubaidr: You are using RNN as your first example, which will suffer from exploding or vanishing gradient, which is why both GRU and LSTM were invented (not by me, but in 1997 by Sepp Hochreiter and Jürgen Schmidhuber).

The variance between GRU and LSTM is simply because GRU tries to be a bit more performant, but will always be very close to LSTM. LSTM, though, has been considered the more stable industry standard for recurrent neural networks.

The reason I can say this with confidence is they all three use the same underlying class with only the formula differing.

Now to the original issue for @fipsi03:

LSTMTimeStep cannot correctly predict a simple array of numbers
It should output 11... (+-0.5)
const training = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];

Consider: If you never told a child about "11", only ever helped him to count to "10", and then one day asked him what was next, what would he say?

The answer: He wouldn't know, much as your net does not.

The problem you are running into is simply because you are not giving the net enough information to know what is next and it is simply over-fitting to the training data. To get better results, simply add more and meaningful training data.

On or more technical note, recurrent, time step, and feed forward neural networks that exist in brain.js are all supervised learning algorithms. There is planning to add in a new unsupervised reinforcement style api's, and we welcome help in suggestions and building them, but as it stands they are not here yet.

[mubaidr]

I was not talking about the difference of results within RNN, GRU or LSTM results. On jsfiddle, running jsfiddle multiple times would sometimes output result dramatically different for the same network. But anyway to the main issue. 👍

[phult]

I was not talking about the difference of results within RNN, GRU or LSTM results. On jsfiddle, running jsfiddle multiple times would sometimes output result dramatically different for the same network. But anyway to the main issue. 👍

How about this issue? Anybody help me? I'm also having this issue.

[robertleeplummerjr]

What version are you on? Is your example the same as is mentioned here, or different. If different, can you share?

[phult]

Hi @robertleeplummerjr, I use version "brain.js": "*"
You can have a look at this example: http://jsfiddle.net/mubaidr/8yeo2jmd/21/
The output results are extremely different after every time i run .forecast()
Thank you

[mubaidr]

^ updated brainjs version here: http://jsfiddle.net/mubaidr/8yeo2jmd/26/

[robertleeplummerjr]

You need to normalize your inputs to values between 0 and 1.

[Theraghav]

Normalised training data works fine:
http://jsfiddle.net/Theraga/z5b2ghc4/53/