New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Masking without embeddings and the new API #816

Closed
sergeyf opened this Issue Oct 11, 2015 · 12 comments

Comments

Projects
None yet
8 participants
@sergeyf

sergeyf commented Oct 11, 2015

Below is my sequence to sequence model (thanks to various folks throughout the forum for helping me assemble this with all of your informative posts). I have 1 dimensional time series, and they are no longer than length 50.

input_dim = 1
seqlen = 50
hidden_size = 16
print('Build model...')
model = Sequential()
model.add(LSTM(input_dim=input_dim, output_dim=hidden_size, return_sequences=False))
model.add(Dense(hidden_size))
model.add(Activation('relu'))
model.add(RepeatVector(seqlen))
model.add(LSTM(hidden_size, return_sequences=True))
model.add(TimeDistributedDense(output_dim=input_dim, activation="linear"))

optimizer = rmsprop(lr=0.01, clipnorm=10)
model.compile(optimizer=optimizer, loss='mse')

This compiles and trains well. The time-series are reproduced very well.

However, I would like to also have masking in the input for the purpose of filling in the inputs. So now I'd like to add a Masking layer, like so:

print('Build model...')
hidden_size = 16
model = Sequential()
model.add(Masking(mask_value=0))
model.add(LSTM(input_dim=input_dim, output_dim=hidden_size, return_sequences=False))
model.add(Dense(hidden_size))
model.add(Activation('relu'))
model.add(RepeatVector(seqlen))
model.add(LSTM(hidden_size, return_sequences=True))
model.add(TimeDistributedDense(output_dim=input_dim, activation="linear"))

optimizer = rmsprop(lr=0.01, clipnorm=10)
model.compile(optimizer=optimizer, loss='mse')

And this provides the traceback:

Build model...
Traceback (most recent call last):

  File "<ipython-input-9-c9d8834ae358>", line 5, in <module>
    model.add(LSTM(input_dim=input_dim, output_dim=hidden_size, return_sequences=False))

  File "build\bdist.win-amd64\egg\keras\layers\containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])

  File "build\bdist.win-amd64\egg\keras\layers\core.py", line 32, in set_previous
    assert self.input_ndim == len(layer.output_shape), "Incompatible shapes: layer expected input with ndim=" +\

  File "build\bdist.win-amd64\egg\keras\layers\core.py", line 82, in output_shape
    return self.input_shape

  File "build\bdist.win-amd64\egg\keras\layers\core.py", line 65, in input_shape
    raise Exception('Layer is not connected. Did you forget to set "input_shape"?')

Exception: Layer is not connected. Did you forget to set "input_shape"

Any ideas on how to add Masking to the model? Thanks very much.

@sergeyf sergeyf changed the title from Using Masking with new API to Using Masking without embeddings and the new API Oct 11, 2015

@sergeyf

This comment has been minimized.

sergeyf commented Oct 11, 2015

I am currently trying this:

model = Sequential()
M = Masking(mask_value=0)
M._input_shape = X.shape
model.add(M)
model.add(LSTM(input_dim=input_dim, output_dim=hidden_size, return_sequences=False))
...

But this seems like a clumsy solution. Does Masking need an input_dim parameter?

@fchollet

This comment has been minimized.

Collaborator

fchollet commented Oct 11, 2015

You simply do:

model = Sequential()
model.add(Masking(mask_value=0, input_shape=(input_dim,)))

All layers support the input_shape keyword argument.

@fchollet fchollet closed this Oct 11, 2015

@sergeyf

This comment has been minimized.

sergeyf commented Oct 12, 2015

Trying that, I get the following error:

Traceback (most recent call last):

  File "<ipython-input-51-d176d93c0584>", line 3, in <module>
    model.add(LSTM(input_dim=input_dim, output_dim=hidden_size, return_sequences=False))

  File "build\bdist.win-amd64\egg\keras\layers\containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])

  File "build\bdist.win-amd64\egg\keras\layers\core.py", line 33, in set_previous
    str(self.input_ndim) + " but previous layer has output_shape " + str(layer.output_shape)

AssertionError: Incompatible shapes: layer expected input with ndim=3 but previous layer has output_shape (None, 1)

I have also tried with model.add(LSTM(output_dim=hidden_size, return_sequences=False)) with a similar error resulting.

@hnykda

This comment has been minimized.

hnykda commented Oct 15, 2015

I am getting same error with 0.2 version and this model:

model = Sequential()  
model.add(LSTM(in_out_neurons, hidden_neurons, return_sequences=False))  
model.add(Dense(hidden_neurons, in_out_neurons))  
model.add(Activation("linear"))  
model.compile(loss="mean_squared_error", optimizer="rmsprop")  

and the error traceback:

Traceback (most recent call last):
  File "keras_vectors.py", line 45, in <module>
    model.add(Dense(hidden_neurons, in_out_neurons))  
  File "/home/dan/prac/.virtualenvs/kerastwo/lib/python3.5/site-packages/keras/layers/containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])
  File "/home/dan/prac/.virtualenvs/kerastwo/lib/python3.5/site-packages/keras/layers/core.py", line 32, in set_previous
    assert self.input_ndim == len(layer.output_shape), "Incompatible shapes: layer expected input with ndim=" +\
  File "/home/dan/prac/.virtualenvs/kerastwo/lib/python3.5/site-packages/keras/layers/recurrent.py", line 41, in output_shape
    input_shape = self.input_shape
  File "/home/dan/prac/.virtualenvs/kerastwo/lib/python3.5/site-packages/keras/layers/core.py", line 65, in input_shape
    raise Exception('Layer is not connected. Did you forget to set "input_shape"?')
Exception: Layer is not connected. Did you forget to set "input_shape"?

adding input_shape like this:

model.add(Dense(hidden_neurons, in_out_neurons, input_shape=(2,)))

doesn't help, then I get an error that int is not callable.

I am pretty sure this worked with older version of Keras and I cannot find the way how it should work with new version.

@holderm

This comment has been minimized.

holderm commented Oct 20, 2015

any updates on this?

@hnykda

This comment has been minimized.

hnykda commented Oct 20, 2015

It's for some reason closed - can you open it again since it seems there are more of us?

@Cospel

This comment has been minimized.

Cospel commented Oct 21, 2015

Older version of keras support this with no problem, but with new version through pip install keras i get same error for LSTM:

Exception: Layer is not connected. Did you forget to set "input_shape"?

@holderm

This comment has been minimized.

holderm commented Oct 21, 2015

Can anyone provice me quick fix how to install the last version with the old API on
a) a "clean" system
b) downgrade from the latest Keras version
That would be great!

I will follow this issue with the new API but I need this particular function for my application, so I'll stick to the old API in the meantime.

@hnykda

This comment has been minimized.

hnykda commented Oct 21, 2015

I guess it shouldn't be discussed here...

But as far as I understand all you need is new virtualenv with pip and install keras using pip install keras==0.1.3 (or whatever version you want). How to do it depends on your system. These keywords should be enough for finding guides and tutorials for your case.

On Linux with Python 3.3+ it's super easy (you need to have non-python dependencies as stated in the README, like HDF5 and so on. For building e.g. numpy you need gcc and so on):

$ virtualenv3 myvenv
$ source myvenv/bin/activate
$ pip install keras==0.1.3
@nicolas-ivanov

This comment has been minimized.

nicolas-ivanov commented Dec 21, 2015

For those people who run into

Exception: Layer is not connected. Did you forget to set "input_shape"?

please notice that in the recent version of keras some interfaces were changed. Before:

LSTM(input_dimention, output_dimension, return_sequences=False)

After:

LSTM(output_dimension, return_sequences=False, input_shape=(num_of_vectors, vect_dimension))

And for the Dense layer, before:

Dense(input_dimention, output_dimension)

After:

Dense(output_dim)

See usage example here.

@shwetgarg

This comment has been minimized.

shwetgarg commented Dec 31, 2015

I am sorry for the trivial question.

I read "http://keras.io/layers/core/" and "code comments" of masking layer. But I am unable to understand the purpose of Masking layer, especially in context of RNN. Also what exactly get_output_mask(train=None) method does in Layer class.

@kevinmartinmauro

This comment has been minimized.

kevinmartinmauro commented Jan 6, 2016

also getting this error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment