Home
This is a summary of the final project for my Artificial Neural Networks course, which I took last semester. A few students (including me) were suggested to learn more about Long Short Term Memory networks and do a little research on them, as a final task. I got interested and decided to take up the challenge.
#Introduction Long Short Term Memory (or LSTM) network is a type of Recurrent Neural Networks, where we introduce memory cell. Check out this article for excellent and very illustrative introduction to LSTM.
#Problem I worked on adding problem (tackled here or here), where input is a sequence of pairs. First component is a real number from the interval [0,1] and the second component is 0 or 1. Exactly two elements are marked with 1. Desired output is the sum of those elements marked with ones.
My research is based on code provided by my professor, Jan Chorowski.
#Goals The ultimate goal would be getting rid of curriculum. A desire is to have a framework that we can use with problems where we cannot introduce curriculum learning.
I've made an attempt to achieve this as well as improve curriculum model.
#My experiments When I understood how this particular architecture works I started tweaking with parameters to get the notion what is the response to them.
As suggested on the lecture by Lukasz Kaiser, in order to 'seal' gates I substituted sigmoid(x) with 1.2sigmoid(x) - 0.1. It helps with gates saturation when it should be either 0 or 1. It turned out to be a major breakthrough. Such change boosted learning process so that it would finish in less than 20000 iterations. It seems that this 'sealing' improves generalization as well.
It turns out that it easier to train network not to return sum of two numbers but its projection on the interval [0,1]. In our case its just average of these two numbers to be added.
(Edit: It turned out that above statement is not true. I overlooked that for given accuracy mean square error in adding problem is four times higher than mean square error in problem with sum projection to interval [0,1].)
Baseline to beat for mean square error would be 0.167 for addition of two numbers or 0.04 for their average. Those are variances of sum or average (i.e. mean square error of that sum is equal to 1) of two i.i.d. variables from uniform distribution on the interval [0,1], respectively.
Two generating methods were used:
- Both ones are in the first half of the sequence and the first is within the first 0.1 length of a sequence.
- First one is randomly placed in the first half of the sequence and the second one is randomly placed in the second half of the sequence.
#Results ###Curriculum learning With use of the first generating method (with projection to [0,1]), curriculum learning (lengths between 10 and 200) the net achieved result presented in semilogarithmic graph below, each test batch contained 2000 examples (Y-axis: mean square error, X-axis: length of sequences in a batch):
As we can see our model generalized for much longer sequences.
Section refers to this code.
After fixing error bound in adding problem (without projection), model with use of the second generating method learns comparably fast. I hoped that maybe this net would generalize so that in problem of adding more that two numbers it would return satisfying results. Unfortunately no such thing happened. However, as previously, it generalized to fit under the baseline for longer sequences. Again I present semilogarithmic graph, each test batch contained 200 examples (Y-axis: mean square error, X-axis: length of sequences in a batch):
###Without curriculum This is much more difficult. I have tried train LSTM net without curriculum on sequences of length 100. Only after 14000 iteration it broke the base line (however significantly).
#Conclusions With curriculum adding problem is moderately easy (with gate sealing). It also generalizes over longer sequences which is good. However, I thing there is still room for improvement (possibly better choice of parameters e.g. learning rate). On the other side, there is a lot of to do in order to completely and effectively get rid of curriculum learning (as long as it is possible with this architecture).