hyperas on LSTM

[parindam]

Hi,

I am using keras 0.3.2 and trying a simple LSTM model with hyperas.
But I get the following error. Anything wrong ?

Exception: Layer is not connected. Did you forget to set "input_shape"?

model = Sequential()
model.add(LSTM(input_dim=388, output_dim=300, return_sequences=False))
model.add(Dropout({{uniform(0, 1)}}))
model.add(Dense(output_dim=1))
model.add(Activation("linear"))
model.compile(loss="mean_squared_error", optimizer="rmsprop")

[maxpumperla]

Hi,

Thanks for reporting this. Would you mind sharing the complete example, so I can examine?

[maxpumperla]

I suspect you get the error, because you don't properly wrap your model into a function. If, in your example, you pass model directly into optim.minimizer, it will only see the first line, i.e.

model = Sequential()

which is clearly lacking input_dim or input_shape. Properly done, it should look as follows:

def return_full_model_here():
    import everything_you_need_for_the_model
    X = ...
    y = ...
    model = Sequential()
    model.add(LSTM(input_dim=388, output_dim=300, return_sequences=False))
    ....
    model.add(Activation("linear"))

    model.compile(loss="mean_squared_error", optimizer="rmsprop")
    model.fit(X, y)
    return {'loss': model_metric_to_minimize, 'status': STATUS_OK}

You need all this, since the model with double brackets dropped-in clearly isn't valid code in any sense and all hyperas does is tell hyperopt what to plug in, so to make this an actual keras model. Therefore the function wrapper.

I hope this makes sense, probably I should update the documentation accordingly.

[parindam]

This is the code i have which is throwing error.
sometimes, it also throws error saying, "OSError: [Errno 2] No such file or directory: './temp.pyc'"
It just works fine without {{...}} and if i call keras_model() directly.

def keras_model():
    from keras.models import Sequential  
    from keras.layers.core import Dense, Activation, Dropout
    from keras.layers.recurrent import LSTM

    from keras.callbacks import EarlyStopping, ModelCheckpoint 


    model = Sequential()  
    model.add(LSTM(input_dim=388, output_dim=300, return_sequences=False)) 
    model.add(Dropout({{uniform(0, 1)}})) 
    model.add(Dense(output_dim=1))  
    model.add(Activation("linear"))  
    model.compile(loss="mean_squared_error", optimizer="rmsprop")

    early_stopping = EarlyStopping(monitor='val_loss', patience=4)
    checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5', 
                                   verbose=1, 
                                   save_best_only=True)

    hist = model.fit(X_train, y_train, 
                     batch_size={{choice([32, 64, 128])}},
                     #batch_size=128,
                     nb_epoch=1, 
                     validation_split=0.08, 
                     show_accuracy=True)
                     #callbacks=[early_stopping, checkpointer])

    score = model.evaluate(X_test, y_test,
                               show_accuracy=True, verbose=0)

    print('Test accuracy:', score[1])
    return {'loss': -score[1], 'status': STATUS_OK}


if __name__ == '__main__':
    best_run = optim.minimize(keras_model,
                              algo=tpe.suggest,
                              max_evals=10,
                              trials=Trials())
    print(best_run)

[jesuisnicolasdavid]

I have the same error as @parindam "OSError: [Errno 2] No such file or directory: './temp.pyc". I am using a virtualenv to run keras. And nothing fancy from the code. Do you have an idea ??

[maxpumperla]

As I don't have access to your data, I modified the keras imdb lstm example and reused every component in your example. It works. Have a try at this:

from __future__ import print_function
from hyperopt import Trials, STATUS_OK, tpe
from hyperas import optim
from hyperas.distributions import choice, uniform


def keras_model():
    from keras.preprocessing import sequence
    from keras.models import Sequential
    from keras.layers.core import Dense, Dropout, Activation
    from keras.layers.embeddings import Embedding
    from keras.layers.recurrent import LSTM
    from keras.datasets import imdb
    from keras.callbacks import EarlyStopping, ModelCheckpoint

    max_features = 20000
    maxlen = 100

    print('Loading data...')
    (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
    print(len(X_train), 'train sequences')
    print(len(X_test), 'test sequences')

    print("Pad sequences (samples x time)")
    X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
    X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
    print('X_train shape:', X_train.shape)
    print('X_test shape:', X_test.shape)

    print('Build model...')
    model = Sequential()
    model.add(Embedding(max_features, 128, input_length=maxlen))
    model.add(LSTM(128))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Dense(1))
    model.add(Activation('sigmoid'))

    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  class_mode="binary")

    early_stopping = EarlyStopping(monitor='val_loss', patience=4)
    checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5',
                                   verbose=1,
                                   save_best_only=True)

    hist = model.fit(X_train, y_train,
                     batch_size={{choice([32, 64, 128])}},
                     # batch_size=128,
                     nb_epoch=1,
                     validation_split=0.08,
                     show_accuracy=True,
                     callbacks=[early_stopping, checkpointer])

    score, acc = model.evaluate(X_test, y_test, show_accuracy=True, verbose=0)

    print('Test accuracy:', acc)
    return {'loss': -acc, 'status': STATUS_OK}

if __name__ == '__main__':
    best_run = optim.minimize(keras_model,
                              algo=tpe.suggest,
                              max_evals=10,
                              trials=Trials())
    print(best_run)

[maxpumperla]

I will publish the above as an example for hyperas. Other than that, I'm afraid right now I'm running out of ideas how to help you.

[jstypka]

@parindam your error might be related with keras rather than hyperas. There are some requirements regarding the first layer in the model, especially if it's a recurrent one and it also depends on the backend you're using. Quote:

TensorFlow warning
For the time being, when using the TensorFlow backend, the number of timesteps used must be specified in your model. Make sure to pass an input_length int argument to your recurrent layer (if it comes first in your model), or to pass a complete input_shape argument to the first layer in your model otherwise.

This might be the issue. Try to have a read here: http://keras.io/layers/recurrent/

[jesuisnicolasdavid]

I copy/paste your new example with LSTM and i have the following error.

IndexError Traceback (most recent call last)
---> 64 trials=Trials())

IndexError: list index out of range

[jstypka]

@jesuisnicolasdavid it works for me. Make sure you use the latest keras version, I'm working on the latest master.

[maxpumperla]

keras 0.3.2 is recommended. Well, "works on my machine". I'm still puzzled why this won't work for you.

[jesuisnicolasdavid]

Ok, i manage to get rid of the previous error with a lot of update. Now i am trying to run the CNN_LSTM example from keras and the error i get is :

IndexError Traceback (most recent call last)
---> 84 trials=Trials())
IndexError: invalid index to scalar variable.

code :

from __future__ import print_function
from hyperopt import Trials, STATUS_OK, tpe
from hyperas import optim
from hyperas.distributions import choice, uniform

def keras_model():

    import numpy as np
    np.random.seed(1337)  # for reproducibility
    from keras.preprocessing import sequence
    from keras.models import Sequential
    from keras.layers.core import Dense, Dropout, Activation
    from keras.layers.embeddings import Embedding
    from keras.layers.recurrent import LSTM, GRU, SimpleRNN
    from keras.layers.convolutional import Convolution1D, MaxPooling1D
    from keras.datasets import imdb

    # Embedding
    max_features = 20000
    maxlen = 100
    embedding_size = 300

    # Convolution
    filter_length = 6
    nb_filter = 64
    pool_length = 4

    # LSTM
    lstm_output_size = 100

    # Training
    batch_size = 60
    nb_epoch = 2

    print('Loading data...')
    (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
    print(len(X_train), 'train sequences')
    print(len(X_test), 'test sequences')

    print('Pad sequences (samples x time)')
    X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
    X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
    print('X_train shape:', X_train.shape)
    print('X_test shape:', X_test.shape)

    model = Sequential()
    model.add(Embedding(max_features, embedding_size, input_length=maxlen))
    model.add(Dropout({{uniform(0, 1)}}))
    model.add(Convolution1D(nb_filter=nb_filter,
                            filter_length=filter_length,
                            border_mode='valid',
                            activation='relu',
                            subsample_length=1))
    model.add(MaxPooling1D(pool_length=pool_length))
    model.add(LSTM(lstm_output_size))
    model.add(Dense(1))
    model.add(Activation('sigmoid'))

    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  class_mode='binary')

    print('Train...')
    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,
              validation_data=(X_test, y_test), show_accuracy=True)
    score, acc = model.evaluate(X_test, y_test, batch_size=batch_size,
                                show_accuracy=True)
    print('Test score:', score)
    print('Test accuracy:', acc)
    return {'loss': -score[1], 'status': STATUS_OK}

if __name__ == '__main__':
    best_run = optim.minimize(keras_model,
                              algo=tpe.suggest,
                              max_evals=10,
                              trials=Trials())
    print(best_run)

[maxpumperla]

Again, this one works for me as well. I will add another example to address it soon. I really want to help, and trust you see that, but I'm not sure this is really the scope of what was issued originally. Any thoughts?

[jesuisnicolasdavid]

For me, the first hyperas evaluation is working well, it is when it passes on the second one that it crashes (just after the print of the score and accuracy). Did you run it for at least two evaluations ? If it is the case, maybe i have another issue with some import. Thanks for the help!

[maxpumperla]

Next time, could you please provide the full stack trace? Thanks.

Traceback (most recent call last):
  File "examples/cnn_lstm.py", line 77, in <module>
    trials=Trials())
  File "/usr/local/lib/python2.7/site-packages/hyperas-0.0.2-py2.7.egg/hyperas/optim.py", line 74, in minimize
    best = fmin(keras_fmin_fnct, space=get_space(), algo=algo, max_evals=max_evals, trials=trials)
  File "/usr/local/lib/python2.7/site-packages/hyperopt/fmin.py", line 334, in fmin
    rval.exhaust()
  File "/usr/local/lib/python2.7/site-packages/hyperopt/fmin.py", line 294, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.async)
  File "/usr/local/lib/python2.7/site-packages/hyperopt/fmin.py", line 268, in run
    self.serial_evaluate()
  File "/usr/local/lib/python2.7/site-packages/hyperopt/fmin.py", line 187, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/usr/local/lib/python2.7/site-packages/hyperopt/fmin.py", line 114, in evaluate
    rval = self.fn(pyll_rval)
  File "./temp_model.py", line 67, in keras_fmin_fnct
IndexError: invalid index to scalar variable.

This indicates an error that arises from treating scalars as iterables.

When you write

score, acc = model.evaluate(X_test, y_test, batch_size=batch_size, show_accuracy=True)

instead of

score = model.evaluate(X_test, y_test, batch_size=batch_size, show_accuracy=True)

then

return {'loss': -score[1], 'status': STATUS_OK}

does not make sense. One of those classic copy and paste things I guess.

[maxpumperla]

I'm going to close the Q&A now. ;)

[jesuisnicolasdavid]

That was a really bad error from me! Thanks for your answer and your help ! You are doing an amazing work!

[catta134]

if I have a model not sequential, how can run hyperopt??