Skip to content
TensorFlow implementation of Independently Recurrent Neural Networks
Branch: master
Clone or download
batzner Merge pull request #15 from batzner/tf1.7-migration
Change import of LayerRNNCell (renamed in TensorFlow 1.7)
Latest commit f7f370f Apr 1, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples Extend documentation for sequential mnist code Mar 27, 2018
img Add sequential mnist results Mar 27, 2018
.gitignore Change code style and documentation slightly Mar 21, 2018
LICENSE Create LICENSE Mar 15, 2018 Update Mar 27, 2018 Change code style and documentation slightly Mar 21, 2018
requirements.txt Change import of renamed LayerRNNCell in TensorFlow 1.7 Mar 31, 2018

Independently Recurrent Neural Networks

Simple TensorFlow implementation of Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN by Shuai Li et al. The author's original implementation in Theano and Lasagne can be found in Sunnydreamrain/IndRNN_Theano_Lasagne.


In IndRNNs, neurons in recurrent layers are independent from each other. The basic RNN calculates the hidden state h with h = act(W * input + U * state + b). IndRNNs use an element-wise vector multiplication u * state meaning each neuron has a single recurrent weight connected to its last hidden state.

The IndRNN

  • can be used efficiently with ReLU activation functions making it easier to stack multiple recurrent layers without saturating gradients
  • allows for better interpretability, as neurons in the same layer are independent from each other
  • prevents vanishing and exploding gradients by regulating each neuron's recurrent weight


Copy into your project.

from ind_rnn_cell import IndRNNCell

# Regulate each neuron's recurrent weight as recommended in the paper
recurrent_max = pow(2, 1 / TIME_STEPS)

cell = MultiRNNCell([IndRNNCell(128, recurrent_max_abs=recurrent_max),
                     IndRNNCell(128, recurrent_max_abs=recurrent_max)])
output, state = tf.nn.dynamic_rnn(cell, input_data, dtype=tf.float32)

Experiments in the paper

Addition Problem

See examples/ for a script reproducing the "Adding Problem" from the paper. Below are the results reproduced with the code.

Sequential MNIST

See examples/ for a script reproducing the Sequential MNIST experiment. I let it run for two days and stopped it after 60,000 training steps with a

  • Training error rate of 0.7%
  • Validation error rate of 1.1%
  • Test error rate of 1.1%


  • Python 3.4+
  • TensorFlow 1.5+
You can’t perform that action at this time.