YamRNN

Yamada modified RNN

Abstract

Recurrent neural networks (RNNs) are among the most promising of the many artificial intelligence techniques now under development, showing great potential for memory, interaction, and linguistic understanding. Among the more sophisticated RNNs are long short-term memory (LSTM) and gated recurrent units (GRUs), which emulate animal brain behavior; these methods yield superior memory and learning speed because of the excellent core structure of their architectures. In this study, we attempted to make further improvements in core structure and develop a novel, compact architecture with a high learning speed. We stochastically generated 30,000 RNN architectures, evaluated their performance, and selected the one most capable of memorizing long contexts with relatively few parameters. This RNN, YamRNN, had fewer parameters than LSTM and GRU by a factor of two-thirds or better and reduced the time required to achieve the same learning performance on a sequence classification task as LSTM and GRU by 80% at maximum. This novel RNN architecture is expected to be useful for addressing problems such as predictions and analyses on contextual data and also suggests that there is room for the development of better architectures.

Implementation of RNNs

YamRNN

Paper: Developing a novel recurrent neural network architecture with fewer parameters and good learning performance

Code for Tensorflow (version 2.0 or later): code/tensorflow/yamrnn.py

Code for Theano: code/theano/yamrnn.py

Elman Network

Paper: Finding structure in time

Code for Tensorflow (version 2.0 or later): code/tensorflow/en.py

Code for Theano: code/theano/en.py

Independently Recurrent Neural Network (IndRNN)

Paper: Independently recurrent neural network (indrnn): Building a longer and deeper rnn

Code for Tensorflow (version 2.0 or later): code/tensorflow/indrnn.py

Code for Theano: code/theano/indrnn.py

LSTM

Paper: Learning to forget: Continual prediction with lstm

Code for Tensorflow (version 2.0 or later): code/tensorflow/lstm.py

Code for Theano: code/theano/lstm.py

Simplified LSTM (S-LSTM)

Paper: Investigating gated recurrent neural networks for speech synthesis

Code for Tensorflow (version 2.0 or later): code/tensorflow/slstm.py

Code for Theano: code/theano/slstm.py

GRU

Paper: Learning phrase representations using rnn encoder-decoder for statistical machine translation

Code for Tensorflow (version 2.0 or later): code/tensorflow/gru.py

Code for Theano: code/theano/gru.py

Minimal Gated Unit (MGU)

Paper: Minimal gated unit for recurrent neural networks

Code for Tensorflow (version 2.0 or later): code/tensorflow/mgu.py

Code for Theano: code/theano/mgu.py

Embedded Gated Recurrent Units (eGRU)

Paper: An optimized recurrent unit for ultra-low-power keyword spotting

Code for Tensorflow (version 2.0 or later): code/tensorflow/egru.py

Code for Theano: code/theano/egru.py

Simple Gated Unit (SGU)

Paper: Deep gate recurrent neural network

Code for Tensorflow (version 2.0 or later): code/tensorflow/sgu.py

Code for Theano: code/theano/sgu.py