How to stack convolutional layer and lstm?

[OnlySang]

I want to extract features through CNN and do squence labeling through LSTM? How could I stack these different layers together? And there would be problem when traning the stacked nn?

59 model = Sequential()
60 model.add(Convolution2D(20, 1, 7, 7, border_mode='full'))
61 model.add(Activation('relu'))
62 model.add(MaxPooling2D(poolsize=(2, 2)))
63 model.add(Convolution2D(50, 20, 5, 5, border_mode='full'))
64 model.add(Activation('relu'))
65 model.add(MaxPooling2D(poolsize=(2, 2)))
66 model.add(Convolution2D(100, 50, 3, 3, border_mode='full'))
67 model.add(Activation('relu'))
68 model.add(MaxPooling2D(poolsize=(2, 2)))
69 model.add(Flatten())
70 #model.add(Dense(100_6_6, 4096, init='normal'))
71 #model.add(Dropout(0.5))
72 model.add(LSTM(100_6_6, 512,return_sequences=True)) # try using a GRU instead, for fun
73 model.add(Dense(512, nb_classes, init='normal'))
74 model.add(Activation('softmax'))
75
76 # try using different optimizers and different optimizer configs
77 sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
78 model.compile(loss='categorical_crossentropy', optimizer=sgd, class_mode="categorical")

[fchollet]

The output of Flatten is going to have the shape (nb_samples, features). The input of LSTM must have the shape (nb_samples, timesteps, features).

What's your input, and what are you actually trying to achieve?

[OnlySang]

@fchollet I know what you say, So I have to add new layers to keras. When it's a convolutional nn, the input is (nb_samples, stack_size, rows, cols); when it's a recurrent nn, the input is (nb_sampels, max_length, features). I stack recurrent after convolution, so the input must be (nb_samples, max_length, stack_size, rows, cols). It's not easy to do this. I will do several reshape between layers. You can learn more about connectionist temporal classification.

[fchollet]

• Put all the pictures in a sequence in a single batch
• insert the following custom layer after your convolution + flatten stage:

class MyReshape(Layer):
    def get_output(self, train):
        X = self.get_input(train)
        nshape = (1,) + X.shape 
        return theano.tensor.reshape(X, nshape)

It turns a batch of N vectors into a batch of size 1 containing a sequence of N vectors.

Note that if you do something like that, the length of the input and of the labels won't match, so you won't be able to use the model.fit() method for training. However, you should be able to use the model.train(X, y) method just fine, on small minibatches.

[simonhughes22]

@fchollet what is the difference between the model.fit and the model.train that you reference above?

[fchollet]

@simonhughes22 it has been renamed train_on_batch. It does a single gradient update on a single batch of data.

[simonhughes22]

Thanks

[usmanatif]

@fchollet hello i followed #4172 #421 and tries every possible way to integrate CNN (VGG16) with LSTM in keras, but i am continuously getting error

below is my code

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
from keras.models import Sequential
from keras.layers import Flatten, LSTM, Dense, TimeDistributed, InputLayer
from keras.layers.core import Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D
import numpy as np

#%%     Image Generator for datasets
datagen = ImageDataGenerator(
        rotation_range = 0,
        width_shift_range = 0,
        height_shift_range = 0,
        shear_range = 0,
        zoom_range = 0,
        rescale = None,
        horizontal_flip = False,
        fill_mode = 'nearest')

train_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Train',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')

test_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Test',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')


#%%     Convolutional Model (Basically Its VGG) + LSTM model (TimeDistributed)

model = Sequential()

model.add(InputLayer(input_shape=(5, 224, 224, 3)))
model.add(TimeDistributed(Convolution2D(64, (3, 3))))
model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))
model.add(LSTM(10))
model.add(Dense(3))

model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])
model.fit_generator(train_generator, epochs=1, steps_per_epoch=len(train_generator.filenames))

i am getting error that (here 5 is time step, 224,224 is image dimension and 3 is channel )
Error when checking input: expected input_2 to have 5 dimensions, but got array with shape (5, 224, 224, 3)

but when i supply samples also
model.add(InputLayer(input_shape=(1, 5, 224, 224, 3)))
i get this error
number of input channels does not match corresponding dimension of filter, 224 != 3

[chineseqsc]

@fchollet hello i followed #4172 #421 and tries every possible way to integrate CNN (VGG16) with LSTM in keras, but i am continuously getting error

below is my code

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
from keras.models import Sequential
from keras.layers import Flatten, LSTM, Dense, TimeDistributed, InputLayer
from keras.layers.core import Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D
import numpy as np

#%%     Image Generator for datasets
datagen = ImageDataGenerator(
        rotation_range = 0,
        width_shift_range = 0,
        height_shift_range = 0,
        shear_range = 0,
        zoom_range = 0,
        rescale = None,
        horizontal_flip = False,
        fill_mode = 'nearest')

train_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Train',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')

test_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Test',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')


#%%     Convolutional Model (Basically Its VGG) + LSTM model (TimeDistributed)

model = Sequential()

model.add(InputLayer(input_shape=(5, 224, 224, 3)))
model.add(TimeDistributed(Convolution2D(64, (3, 3))))
model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))
model.add(LSTM(10))
model.add(Dense(3))

model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])
model.fit_generator(train_generator, epochs=1, steps_per_epoch=len(train_generator.filenames))

i am getting error that (here 5 is time step, 224,224 is image dimension and 3 is channel )
Error when checking input: expected input_2 to have 5 dimensions, but got array with shape (5, 224, 224, 3)

but when i supply samples also
model.add(InputLayer(input_shape=(1, 5, 224, 224, 3)))
i get this error
number of input channels does not match corresponding dimension of filter, 224 != 3

I met this error too,have you solved it?

[aezco]

@fchollet hello i followed #4172 #421 and tries every possible way to integrate CNN (VGG16) with LSTM in keras, but i am continuously getting error

below is my code

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
from keras.models import Sequential
from keras.layers import Flatten, LSTM, Dense, TimeDistributed, InputLayer
from keras.layers.core import Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D
import numpy as np

#%%     Image Generator for datasets
datagen = ImageDataGenerator(
        rotation_range = 0,
        width_shift_range = 0,
        height_shift_range = 0,
        shear_range = 0,
        zoom_range = 0,
        rescale = None,
        horizontal_flip = False,
        fill_mode = 'nearest')

train_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Train',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')

test_generator = datagen.flow_from_directory(
        'D:\SpyderDeepLearning\PersonReidentification\Dataset_Light\Test',
        target_size = (224,224),
        batch_size = 1, 
        class_mode = 'categorical')


#%%     Convolutional Model (Basically Its VGG) + LSTM model (TimeDistributed)

model = Sequential()

model.add(InputLayer(input_shape=(5, 224, 224, 3)))
model.add(TimeDistributed(Convolution2D(64, (3, 3))))
model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))
model.add(LSTM(10))
model.add(Dense(3))

model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])
model.fit_generator(train_generator, epochs=1, steps_per_epoch=len(train_generator.filenames))

i am getting error that (here 5 is time step, 224,224 is image dimension and 3 is channel )
Error when checking input: expected input_2 to have 5 dimensions, but got array with shape (5, 224, 224, 3)

but when i supply samples also
model.add(InputLayer(input_shape=(1, 5, 224, 224, 3)))
i get this error
number of input channels does not match corresponding dimension of filter, 224 != 3

@fchollet @chineseqsc @usmanatif
I am having the same problem, did you guys solve it?