Recurrent Neural Networks

This repository contains the code for Recurrent Neural Network from scratch using Python 3 and numpy.

Requirements

Numpy

Sample Output

Error:[ 3.94375112]
Pred:[1 1 1 1 1 1 1 1]
True:[0 0 1 1 0 1 1 1]
28 + 27 = 255
------------
Error:[ 3.89378112]
Pred:[0 0 1 0 1 0 1 0]
True:[1 0 0 1 0 0 1 0]
21 + 125 = 42
------------
Error:[ 3.80079469]
Pred:[1 1 1 0 1 0 0 1]
True:[1 1 0 1 1 0 0 1]
100 + 117 = 233
------------
Error:[ 3.75256184]
Pred:[0 0 0 1 0 0 0 0]
True:[0 0 1 0 0 1 1 1]
12 + 27 = 16
------------
Error:[ 3.47163732]
Pred:[0 0 0 1 0 1 0 1]
True:[0 1 0 1 1 1 0 1]
67 + 26 = 21
------------
Error:[ 3.46614289]
Pred:[1 1 1 0 0 1 1 0]
True:[1 0 0 0 0 1 1 0]
92 + 42 = 230
------------
Error:[ 0.57723326]
Pred:[0 1 1 1 0 1 1 0]
True:[0 1 1 1 0 1 1 0]
86 + 32 = 118
------------
Error:[ 0.83430643]
Pred:[1 1 1 0 1 0 1 0]
True:[1 1 1 0 1 0 1 0]
107 + 127 = 234
------------
Error:[ 0.50010502]
Pred:[0 0 1 0 1 0 0 0]
True:[0 0 1 0 1 0 0 0]
15 + 25 = 40
------------
Error:[ 0.42438922]
Pred:[0 1 1 1 0 1 1 1]
True:[0 1 1 1 0 1 1 1]
28 + 91 = 119
------------

Note that the RNN keeps on training, predicting output values and collecting dJdW2 and dJdW1 values at each output stage. Once it reaches the last stage of an addition, it starts backpropagating all the errors till the first stage. Hence, after initial 3-4 steps it starts predicting the accurate output.

Resources

S.No.	Papers / Blogs / Authors	Paper Links
1.	"A Critical Review of RNN for Sequence Learning" by Zachary C. Lipton	https://arxiv.org/pdf/1506.00019.pdf
2.	"i am trask" Blog	https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/
3.	Nikhil Buduma's Blog	http://nikhilbuduma.com/2015/01/11/a-deep-dive-into-recurrent-neural-networks/