How to add features elementwise to separate GRU

[kgruhler]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Hi all,

I am new to ML and Keras and I have a question about the connection between CNN and RNN.
My current network has 3 CNN- and 1 RNN (GRU) -Layers. Data is TimeDistributed, but I think it doesn't matter. My model looks like the following:

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (5, 3), kernel_initializer="he_normal", activation='relu'), input_shape=self.input_shape))
model.add(TimeDistributed(Conv2D(32, (3, 3), kernel_initializer="he_normal", activation='relu')))
model.add(TimeDistributed(Flatten()))
model.add(GRU(128))
model.add(Flatten())
model.add(Dense(512))
model.add(Dropout(0.5))
model.add(Dense(self.nb_classes, activation='softmax'))

I'm trying to add the features separately into a GRU-Unit for each to get a feature-vector over time for each feature (see image below (red frame)). Now, the only thing I have is a feature-vector of 32 flatted features, which are all feed into one GRU-Layer (see image below ("current situation")).

Is there anything that I can do between the CNN and RNN to get the desired behavior?

Thank you so much in advance.

[PascalLiebl]

To solve your issue you have to use the Keras' functional API (https://keras.io/getting-started/functional-api-guide/) to be able to split a layer output and define which input the model should use for your following layers.
An example for this would be the following (and excuse me for using my dimensions):

input = Input(shape=(400, 65, 17))   # 400 frames of 65x17 image
time_distr_conv = TimeDistributed(Conv2D(4, (3, 3), kernel_initializer='he_normal', activation='relu'))(input)
flatten_conv = TimeDistributed(Flatten())(time_distr_conv)

first_quarter = crop(2, 0, 945)(flatten_conv)
second_quarter = crop(2, 945, 1890)(flatten_conv)
third_quarter = crop(2, 1890, 2835)(flatten_conv)
fourth_quarter = crop(2, 2835, 3780)(flatten_conv)

gru1 = GRU(256)(first_quarter)
gru2 = GRU(256)(second_quarter)
gru3 = GRU(256)(third_quarter)
gru4 = GRU(256)(fourth_quarter)

merge_grus = keras.layers.concatenate([gru1, gru2, gru3, gru4])
dense_merge = Dense(256, kernel_initializer='normal', activation='relu')(merge_grus)  
... 
model = Model(inputs=[input], outputs=output)
model.summary()

marc-moreaux commented his crop function on this issue: #890 (thanks btw!)

def crop(dimension, start, end):
    # Crops (or slices) a Tensor on a given dimension from start to end
    # example : to crop tensor x[:, :, 5:10]
    # call slice(2, 5, 10) as you want to crop on the second dimension
    def func(x):
        if dimension == 0:
            return x[start: end]
        if dimension == 1:
            return x[:, start: end]
        if dimension == 2:
            return x[:, :, start: end]
        if dimension == 3:
            return x[:, :, :, start: end]
        if dimension == 4:
            return x[:, :, :, :, start: end]
    return Lambda(func)

In order to make this approach work for your task, you simply have to adjust the dimensions used in the input and the gru variables.
I hope this helps and let me know if this works for you!

Cheers,
Pascal