Tensorflow compilation takes too long and eats memory

[michiexile]

I try to get up and running with Keras+Tensorflow for RNN for text analysis. I have managed to push my problem down to the following:

mvj@course-03:~/reddit$ ipython
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.

In [1]: import keras, keras.layers.core, keras.layers.recurrent, keras.models, tensorflow
Using TensorFlow backend.
In [2]: model = keras.models.Sequential()
In [3]: model.add(keras.layers.core.Lambda(lambda c: c, input_shape=(6486, 97), output_shape=(6486, 97)))
In [4]: model.add(keras.layers.recurrent.LSTM(12))
In [5]: model.summary()
--------------------------------------------------------------------------------
Initial input shape: (None, 6486, 97)
--------------------------------------------------------------------------------
Layer (name)                  Output Shape                  Param #             
--------------------------------------------------------------------------------
Lambda (lambda)               (None, 6486, 97)              0                   
LSTM (lstm)                   (None, 12)                    5280                
--------------------------------------------------------------------------------
Total params: 5280
--------------------------------------------------------------------------------
In [6]: model.compile(loss='MAE', optimizer='rmsprop')

At this point, the machine starts churning with a single processor pegged at 100%, and allocating memory. I stopped this once it got up to several gigs (4-5 or so).

Please make sure that the boxes below are checked before you submit your issue. Thank you!

• Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps
• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[fchollet]

It looks like you are trying to compile a LSTM with 6486 timesteps. In TensorFlow, LSTMs (and other RNNs) are unrolled, and so you are going to generate 6.5k timesteps worth of graph operations. Obviously this is not going to work out well.

Solution: break your sequences into smaller ones, and maybe use a stateful LSTM if you think that's necessary. I recommend sequences with at most 200 timesteps.