Blank output when input string.length > ~100 LSTM/GRU

[1mike12]

What is wrong?

I can get the toy examples working like the happy, sad example on the documentation. But when I increase the length of the sentence, I start getting blank strings as output "". Through experimentation it seems to have to do with sentence length and happens at around 100 characters.

Where does it happen?

running node 10.7 on pc and mac

How do we replicate the issue?

run this code as is to see working example.
Then swap out the commented variable happySentence and see that the outputs become blank string

The callback outputs will start to get the right answer really quickly; around 150 iterations. But nothing ever gets output in the longer sentence, even when approaching tens of thousands of iterations

const net = new brain.recurrent.GRU();
        const happySentence = "is upset that he can't update his Facebook by texting it... and might cry as a result School"
        // const happySentence = 'is upset that he can\'t update his Facebook by texting it... and might cry as a result School Today'
        console.log(happySentence.length)
        const sadSentence = 'im meeting up with one of my besties tonight! Cant wait!! - GIRL TALK!!'
        net.train([
                {input: happySentence, output: [1]},
                {input: sadSentence, output: [0]},
            ],
            {
                iterations: 20000,    // the maximum times to iterate the training data --> number greater than 0
                errorThresh: 0.005,   // the acceptable error percentage from training data --> number between 0 and 1
                log: true,           // true to use console.log, when a function is supplied it is used --> Either true or a function
                logPeriod: 25,        // iterations between logging out --> number greater than 0
                learningRate: .1,    // scales with delta to effect training rate --> number between 0 and 1
                momentum: .1,        // scales with next layer's change value --> number between 0 and 1
                callback: function(){
                    console.log(net.run(happySentence))
                },       // a periodic call back that can be triggered while training --> null or function
                callbackPeriod: 25,   // the number of iterations through the training data between callback calls --> number greater than 0
                timeout: Infinity     // the max number of milliseconds to train for --> number greater than 0
            });

        const output = net.run(happySentence);
        expect(output).equal(1)

How important is this (1-5)?

5 because it doesn't work

Expected behavior (i.e. solution)

the output of either versions of happySentence should have been 1

Other Comments

[ewliang]

I'm getting the same issue too when using LSTM. I'm trying to make a classification tool, and I get blank outputs often. I tried high learning rate, low learning rate, high iteration, low iteration, and even experimented with the data input format for the training in all the possible combinations listed.

[1mike12]

@ewliang same. After looking into the source code of rnn.js, I tried other parameters for the constructor that weren't listed on the documentation. I hoped that maybe bumping input size and output size to 256 could have something to do with it, but it didn't work.

[robertleeplummerjr]

I believe the droids you are looking for are here: https://github.com/BrainJS/brain.js/blob/master/src/recurrent/rnn.js#L316

net.run(rawInput = [], maxPredictionLength = 100, isSampleI = false, temperature = 1)

I think the original idea was to prevent the net from thinking forever. We're open to recommendations to make this feature better.

[robertleeplummerjr]

Clarifying, you want to make maxPredictionLength larger than 100.

[robertleeplummerjr]

Working: https://jsfiddle.net/robertleeplummerjr/xw68qesp/6/

We probably should document this before this issue closes.

[1mike12]

ok that's awesome! Could you go into a bit as to how that parameter affects the model? Like if I were to guess from the name of the variable, it would cap the length of the final output. But in our cases of classification, the lengths of the outputs are trivially small.

[robertleeplummerjr]

All it does it cap the output length so to prevent the recurrent net from going into a loop that never ends. I suppose the length should really be after the length of the input that you send it, in which case, this is a bug.

[robertleeplummerjr]

After reading this a second time, I don't think I answered the question asked last.

The question:

Could you go into a bit as to how that parameter affects the model?

The parameter maxPredictionLength has to do with the repeating (recurrent) part of the hidden layers of the model. The hidden layers repeat internally but are all connected to the same context that tracks their behaviour and what to expect next. This recurrent area of the model connects from the input, over and over again to itself with newly input data, and then to the output layer.

The network is a self governing engine. At first it isn't smart enough to self govern. If you can imaging an engine that isn't smart, but which has unfettered access to the throttle...

Without maxPredictionLength the engine (net) can open the throttle when unaided (unaided = predicting. Aided = training) resulting in a catastrophe where the engine explodes (net w/ never ending loop). By simply capping the maxPredictionLength, or rather telling the motor "Whoa nelly!" when the net isn't being aided (predicting), we're simply looking after yall.

This is a fairly common practice, and is one that we borrowed from recurrent.js, but is being used to construct the GPU architecture that is coming in v2.

[robertleeplummerjr]

Now you can change this with net.maxPredictionLength = number. We'll add this into the docs asap.