Multiple Inputs

[benjaminklein]

Hi, Any plans to add support for multiple inputs?

For example - Having two images - each will be passed through a different CNN and the final representations of the two will be merged at the end.

Thank you!

[fchollet]

We added that some time ago. See the Merge layer: http://keras.io/layers/core/#merge

Note that in concat mode, on GPU, you will have to use the latest version of Theano (get it from the repo, not pip), due a bug with past Theano versions.

[benjaminklein]

Not sure that this is what I wanted: Let say that my input contains two images. I would like one to be passed through CNN1 and the second to be passed through CNN2. Then I can merge them using the merge layer. But how can I use the library in order to handle the two different inputs? Basically I would like to have more than one input or to be able to split the input (using a split layer) to a few layers.. so each sub-input could be passed to a different network...

[fchollet]

That's what the Merge layer allows you to do. You can have two different networks, with different inputs (like 2 images) and you can merge their output into a single tensor.

[benjaminklein]

So let say that I want to train such a model.
I have two networks: CNN1 and CNN2.

My training data contain pairs of images: (i1, i2). I want that i1 will be passed through CNN1 and i2 will be passed through CNN2.

How should I design the network and how should I perform the training such that CNN1 will be applied only on i1 and CNN2 will be applied only on i2?

Thank you.

[fchollet]

The code snippet example in the doc page I linked provides all the info you need.

You will train your model with list of inputs:

model.fit([X_CNN1, X_CNN2], y) 

Which will feed at time t X_CNN1[t] to CNN1 and X_CNN2[t] to CNN2.

[benjaminklein]

Thank you!

[karishmamalkan]

I had a similar requirement. What if i need to pass two inputs into the same node. But these two inputs aren't of the same dimension. Say 1 is an image, and the other is some hidden representation of another image in a lower dimension. I cannot concatenate them since they are of different dimensions. Is there any support to access two totally different inputs within a layer. ( i am modifying keras source to add a new node type, LSTM2. But I only have access to one input, i.e. x)

[jwgu]

@karishmamalkan I need this feature too. Did you get any update on this?

[karishmamalkan]

@jwgu Hi, There was no method to pass multiple inputs to the RNN except to concatenate,dot etc as described by the merge layer. But i found a work around. If you want two inputs, both of which need to be multiplied by trainable weights, then you can use a Graph layer as follows:

Supposed you have two inputs x1 and x2 at each step of the RNN/LSTM. Your RNN function looks like:
h(t) = (Wh * h(t-1) + W1 * x1 + W2 *x2),

then you can have a

1. Dense layer to perform (W1 * x1 +b1) --->Dense1
2. Dense layer to perform (W2 * x2 +b2) --->Dense2
3. Merge Layer to sum Dense1 and Dense2, so you get: (W1 * x1 + W2 *x2)

and then you can pass this sequence into the RNN layer.

[ShunyuanZ]

Supposedly this Merge Layer should support merging outputs from multiple (>2) sources, right?
I tried concatenating 3 layers, however it failed...
Here is the code for merging:

from keras.layers import Merge

left_branch = Sequential()
left_branch.add(Dense(32, input_dim=784))

middle_branch = Sequential()
middle_branch.add(Dense(32, input_dim=784))

right_branch = Sequential()
right_branch.add(Dense(32, input_dim=784))

merged = Merge([left_branch, middle, right_branch], mode='concat')

final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(10, activation='softmax'))

Here is the code for model fit (of course model compiled before):
final_model.fit([X_left, X_middle, X_right], y)

But I get the following error:

Traceback (most recent call last):
  File "MyCNN_multiple_Img.py", line 170, in <module>
    callbacks=[TensorBoard(log_dir=logpath,histogram_freq=1,write_graph=False)]
  File "/usr/bin/lib/python2.7/site-packages/keras/models.py", line 429, in fit
    sample_weight=sample_weight)
  File "/usr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 1036, in fit
    batch_size=batch_size)
  File "/usr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 963, in _standardize_user_data
    exception_prefix='model input')
  File "/husr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 51, in standardize_input_data
    '...')
Exception: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 arrays but instead got the following list of 3 arrays: [array([[[[ 0.49803922,  0.42352942,  0.36862746, ...,  0.68627453,
           0.83529413,  0.85490197],
         [ 0.87843138,  0.98431373,  1.        , ...,  0.4509804 ,
           0.42352942,  0.44...

Is it possible to merge more than two outputs? If so, could some one tell me how should I change my code? Thank you!

[fchollet]

It is possible to merge three tensors, basically just the way you describe but without the syntax errors. You are not sharing the code that you are actually using (it wouldn't even run!), but the code you are actually using presumably involves only two inputs. The following code runs fine:

from keras.models import Sequential
from keras.layers import Dense, Merge

left_branch = Sequential()
left_branch.add(Dense(32, input_dim=784))

middle_branch = Sequential()
middle_branch.add(Dense(32, input_dim=784))

right_branch = Sequential()
right_branch.add(Dense(32, input_dim=784))

merged = Merge([left_branch, middle_branch, right_branch], mode='concat')

final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(10, activation='softmax'))

print final_model.inputs

[ShunyuanZ]

Thank you very much @fchollet !
Sorry didn't put my full code on since it's too long...

But now I think I've figured out what the problem was:
Since in my case, both left_branch and middle branch consist of a stack of convolution+padding+max-pooling layers (copied VGG16 architecture), there is a long code for constructing such a sequential model. So, for laziness...I constructed the sequential model for only once, specifically, I did the following:

from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D, ZeroPadding2D
model = Sequential()
model.add(ZeroPadding2D((1, 1), input_shape=(3, 224, 224)))

model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

Then I say:

left_branch=model
left_branch.add(Flatten(input_shape=model.output_shape[1:]))
left_branch.add(Dense(output_dim=4096, activation='relu'))

middle_branch=left_branch  ## Here is for laziness

right_branch = Sequential()
right_branch.add(Dense(output_dim=400, input_dim=200,activation='linear'))

So far, I thought I've constructed the two branches, with left and middle branching having the same structure. So I merge them

from keras.layers import Merge
merged = Merge([left_branch, middle_branch, right_branch], mode='concat')
final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(output_dim=1, activation='sigmoid'))

Then I compile and train the model:

final_model.compile(loss='mean_squared_error', optimizer='adadelta', metrics=['accuracy'])
from keras.callbacks import TensorBoard
logpath=mypath+'/mylog/log1'
final_model.fit([img_left, img_middle, text_right], y,
          nb_epoch=60, batch_size=20, shuffle=True,
                callbacks=[TensorBoard(log_dir=logpath,histogram_freq=1,write_graph=False)] )

And the following error raised:
Traceback (most recent call last):
  File "MyCNN_multiple_Img.py", line 170, in <module>
    callbacks=[TensorBoard(log_dir=logpath,histogram_freq=1,write_graph=False)]
  File "/usr/bin/lib/python2.7/site-packages/keras/models.py", line 429, in fit
    sample_weight=sample_weight)
  File "/usr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 1036, in fit
    batch_size=batch_size)
  File "/usr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 963, in _standardize_user_data
    exception_prefix='model input')
  File "/husr/bin/lib/python2.7/site-packages/keras/engine/training.py", line 51, in standardize_input_data
    '...')
Exception: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 arrays but instead got the following list of 3 arrays: [array([[[[ 0.49803922,  0.42352942,  0.36862746, ...,  0.68627453,
           0.83529413,  0.85490197],
         [ 0.87843138,  0.98431373,  1.        , ...,  0.4509804 ,
           0.42352942,  0.44...

I then printed the input shape of final_model and got:
[(None, 3, 224, 224), (None, 200)]
while I was expecting:
[(None, 3, 224, 224), (None, 3, 224, 224), (None, 200)]

---

So I assume a safe way of merging multiple sources is to, for each source, go from the very beginning towards the end to construct a sequential model, then merge them (after I did so, the code worked)?
If we have some branches that have the same configuration/structure, is it possible to go through the construction only once, for all those branches (the best I can come up with is to make a function of constructing the sequential model...)? Thanks!

[ersinyar]

@fchollet regarding your answer
The code snippet example in the doc page I linked provides all the info you need.

You will train your model with list of inputs:

model.fit([X_CNN1, X_CNN2], y)
Which will feed at time t X_CNN1[t] to CNN1 and X_CNN2[t] to CNN2.

Does model.fit accepts multiple inputs for validation data? For example, Is the following line legitimate?
model.fit([X_CNN1, X_CNN2], y, .... , validation_data=([X_CNN1_val, X_CNN2_val], y_val))

[monaj07]

@karishmamalkan Thanks for your tip on how to provide two different inputs for an RNN. However in this way, later on, a new set of weights are applied to the merged input by the recurrent class, is that right?
Is it fine to have these extra set of parameters in the network?
Sorry if my question sounds naive, as I am new to Keras and in general, to deep learning.

[wddabc]

Hi sorry for hacking into this thread. I currently need to implement a layer that needs two inputs but has its own trainable parameters. For example, a linear-chain CRF layer takes tokens and tags as inputs, but it has its own trainable parameters (emission matrix and transition matrix). In this case, should I write a layer that is inherited from Merge?

[fatemaaa]

Hi benjaminklein,
I need to ask you if you found any answer for your question because I need to do the same thing but I don't know how.

[SinghGauravKumar]

@fchollet @benjaminklein How does the problem of 2 CNNs is solved in Keras 2? @fchollet : The link doesn't contain any code snippet anymore.
@ersinyar @fatemaaa Have you people figured it out already?

[sajjansh]

@fchollet: Can we merge layers with inputs of different sizes ? For example, I would like to merge two LSTM layers, first layer with input sequence shape (30,2) and the other layer with input sequence shape (15,1). I would then like to merge these two layers and pass the output to a second layer.

Please let me know !

[ShivamPanchal]

I am trying to add three models on passenger data, below is code. I am getting the following error.

OUTPUT
[<tf.Tensor 'dense_249_input:0' shape=(?, 15) dtype=float32>, <tf.Tensor 'conv1d_50_input:0' shape=(?, 15, 1) dtype=float32>, <tf.Tensor 'lstm_27_input:0' shape=(?, 15, 1) dtype=float32>]
(80, 15)
(80, 15, 1)
(80, 15, 1)
ValueError: Error when checking target: expected dense_258 to have shape (None, 10) but got array with shape (80, 1)

`
import pandas as pd
import math
import numpy as np
from pandas import read_csv
from pandas import datetime
from pandas import Series
from pandas import DataFrame
from keras.models import Sequential
from keras.layers import Dense, Merge
from keras.layers import LSTM, Dropout, SimpleRNN
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from keras.layers import Convolution1D, Conv2D, Dense, Flatten
from keras import optimizers
from keras import losses
from keras.layers.pooling import MaxPooling1D
from keras.layers.convolutional import ZeroPadding1D

convert an array of values into a dataset matrix

def create_dataset(dataset, look_back):
dataX, dataY = [], []
for i in range(len(dataset)-look_back-1):
a = dataset[i:(i+look_back), 0]
dataX.append(a)
dataY.append(dataset[i + look_back, 0])
return numpy.array(dataX), numpy.array(dataY)

fix random seed for reproducibility

numpy.random.seed(7)

load the dataset

dataframe = pd.read_csv('/home/shivampanchal/PycharmProjects/WeatherPrediction/data/pass.csv')

convert into datatime

dataframe['yy_mnh']=pd.to_datetime(dataframe['Month'])
dataframe=dataframe.drop('Month',axis=1)
dataframe=dataframe.sort_values(by=['yy_mnh'])
dataframe=dataframe.drop('yy_mnh',axis=1)
dataset = dataframe.values
dataset = dataset.astype('float32')

normalize the dataset

scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)

split into train and test sets

train_size = int(len(dataset) * 0.67)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

reshape into X=t and Y=t+1

look_back = 15

trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)

#NN
trainX_NN = np.reshape(trainX, (trainX.shape[0], trainX.shape[1]))
testX_NN = np.reshape(testX, (testX.shape[0], testX.shape[1]))

#CNN
trainX_CNN = np.reshape(trainX, (trainX.shape[0], trainX.shape[1], 1))
testX_CNN = np.reshape(testX, (testX.shape[0], testX.shape[1], 1))

#LSTM
trainX_LSTM = numpy.reshape(trainX, (trainX.shape[0], look_back , 1))
testX_LSTM = numpy.reshape(testX, (testX.shape[0], look_back, 1))

batch_size = 5
######################
model1 = Sequential()
model1.add(Dense(10, input_dim=trainX_NN.shape[1], activation='relu'))
model1.add(Dense(8, activation='relu'))
model1.add(Dense(1))
######################
model2 = Sequential()
#model2.add(ZeroPadding1D(padding=1, input_shape=(trainX_CNN.shape[1], 1)))
model2.add(Convolution1D(40, 2, input_shape=(trainX_CNN.shape[1], 1), activation='relu'))
#model2.add(ZeroPadding1D(padding=1))
model2.add(Convolution1D(20, 3, activation='relu'))
model2.add(Flatten())
model2.add(Dense(10))
model2.add(Dense(5))
model2.add(Dense(1))
######################
model3 = Sequential()
model3.add(LSTM(32, input_shape=(look_back, 1),activation='relu'))
model3.add(Dense(16))
model3.add(Dense(8))
model3.add(Dense(1))

merged = Merge([model1, model2, model3], mode='concat')

final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(10, activation='relu'))

print final_model.inputs
final_model.compile(optimizer='adam',
loss='mean_squared_error')

print(trainX_NN.shape)
print(trainX_CNN.shape)
print(trainX_LSTM.shape)

final_model.fit([trainX_NN, trainX_CNN, trainX_LSTM], trainY, epochs=10, batch_size=batch_size, verbose=2)
`

[jamartinh]

This works well.

validation_data=({'daily_input': X_test,
                  'constant_input':X_test_constant}, 
                 {'classify_output': Y_test})


training_data = ({'daily_input': X_train,
                  'constant_input':X_train_constant},
                 {'classify_output': Y_train})

model.fit(
    training_data[0],
    training_data[1],
    epochs=900,
    batch_size=batch_size,
    verbose = 1,
    validation_data=validation_data,
    class_weight = class_weight)

Perhaps adding a "data" parameter as an option to define "X" and "y", could make this even more clear and consistent and thus we can use the "training_data " dict as the sole input.

[vinayakumarr]

I have two images, first image and its label is good, second images and its label is bad. I want to pass both images at a time to deep learning model for training. While testing I will have two images (unlabelled) and I want to detect which one is good and which one is bad. Could you please tell how to do it?

[scstu]

I built an architecture using merge and tried to fit the model:

# fine-tune the model history = model.fit_generator( [train_generator1, train_generator2], steps_per_epoch=nb_train_samples//32, epochs=nb_epoch, validation_data=[validation_generator1,validation_generator2], validation_steps = nb_validation_samples//32, callbacks=[reduce_lr])

but got the following error

`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
18 validation_data=[validation_generator1,validation_generator2],
19 validation_steps = nb_validation_samples//32,
---> 20 callbacks=[reduce_lr])
21
22

/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
89 warnings.warn('Update your ' + object_name + 90 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1254 use_multiprocessing=use_multiprocessing,
1255 shuffle=shuffle,
-> 1256 initial_epoch=initial_epoch)
1257
1258 @interfaces.legacy_generator_methods_support

/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
89 warnings.warn('Update your ' + object_name + 90 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper

/usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2114 str(validation_data))
2115 val_x, val_y, val_sample_weights = self._standardize_user_data(
-> 2116 val_x, val_y, val_sample_weight)
2117 val_data = val_x + val_y + val_sample_weights
2118 if self.uses_learning_phase and not isinstance(K.learning_phase(), int):

/usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in _standardize_user_data(self, x, y, sample_weight, class_weight, check_batch_axis, batch_size)
1424 self._feed_input_shapes,
1425 check_batch_axis=False,
-> 1426 exception_prefix='input')
1427 y = _standardize_input_data(y, self._feed_output_names,
1428 output_shapes,

/usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in _standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
71 else:
72 data = data.values if data.class.name == 'DataFrame' else data
---> 73 data = [np.expand_dims(data, 1)] if data.ndim == 1 else [data]
74
75 if len(data) != len(names):

AttributeError: 'DirectoryIterator' object has no attribute 'ndim'`

anyone please tell me whats wrong ?

@fchollet @benjaminklein @karishmamalkan @jamartinh

[Neehal-Lingayat]

Hi, i have similar problem with this issue. i want to merge a thermal and visible image features using deep learning. and i tried to train model u have explained.
model.fit([X_CNN1, X_CNN2], y)
but it shows me the following error:

Traceback*******************
IndexError Traceback (most recent call last)
in ()
----> 1 final_model.fit([train_x1,train_x2],[train_y1,train_y2],steps_per_epoch=20,epochs=10,verbose=1)

c:\users\neehal lingayat\anaconda3\envs\tensorflow1\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
953 sample_weight=sample_weight,
954 class_weight=class_weight,
--> 955 batch_size=batch_size)
956 # Prepare validation data.
957 do_validation = False

c:\users\neehal lingayat\anaconda3\envs\tensorflow1\lib\site-packages\keras\engine\training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, check_array_lengths, batch_size)
790 feed_output_shapes,
791 check_batch_axis=False, # Don't enforce the batch size.
--> 792 exception_prefix='target')
793
794 # Generate sample-wise weight values given the sample_weight and

c:\users\neehal lingayat\anaconda3\envs\tensorflow1\lib\site-packages\keras\engine\training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
88 data = data.values if data.class.name == 'DataFrame' else data
89 data = [data]
---> 90 data = [standardize_single_array(x) for x in data]
91
92 if len(data) != len(names):

c:\users\neehal lingayat\anaconda3\envs\tensorflow1\lib\site-packages\keras\engine\training_utils.py in (.0)
88 data = data.values if data.class.name == 'DataFrame' else data
89 data = [data]
---> 90 data = [standardize_single_array(x) for x in data]
91
92 if len(data) != len(names):

c:\users\neehal lingayat\anaconda3\envs\tensorflow1\lib\site-packages\keras\engine\training_utils.py in standardize_single_array(x)
17 elif K.is_tensor(x):
18 shape = K.int_shape(x)
---> 19 if shape is None or shape[0] is None:
20 raise ValueError(
21 'When feeding symbolic tensors to a model, we expect the'

IndexError: tuple index out of range

---

my full coded is as follows:
CODE*************************
from keras.layers import Activation
from keras.models import Sequential
from keras.optimizers import SGD,Adam
from keras.layers import Dense, Input,Conv2D,MaxPooling2D,Dropout
from keras.layers.core import Flatten
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy
import numpy as np
from keras.models import load_model
from keras.datasets import mnist
from keras.models import Model
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
import pandas
from pandas import DataFrame
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

​
data_path1 = 'C:\tensorflow1\models\research\object_detection\images\csv1.tfrecords' # address to save the hdf5 file
with tf.Session() as sess:
feature = {'train/image': tf.FixedLenFeature([], tf.string),
'train/label': tf.FixedLenFeature([], tf.int64)}
# Create a list of filenames and pass it to a queue
filename_queue = tf.train.string_input_producer([data_path1], num_epochs=1)
# Define a reader and read the next record
reader = tf.TFRecordReader()
_, serialized_example = reader.read(filename_queue)
# Decode the record read by the reader
features = tf.parse_single_example(serialized_example, features=feature)
# Convert the image data from string back to the numbers
image = tf.decode_raw(features['train/image'], tf.float32)

# Cast label data into int32
label = tf.cast(features['train/label'], tf.int32)
# Reshape image data into the original shape
image = tf.reshape(image, [150, 150, 1])
images, labels = tf.train.shuffle_batch([image, label], batch_size=10, capacity=30, num_threads=1, min_after_dequeue=10) 

train_x1=image
train_y1=label
​

​
data_path2 = 'C:\tensorflow1\models\research\object_detection\images\csv2.tfrecords' # address to save the hdf5 file
with tf.Session() as sess:
feature = {'train/image': tf.FixedLenFeature([], tf.string),
'train/label': tf.FixedLenFeature([], tf.int64)}
# Create a list of filenames and pass it to a queue
filename_queue = tf.train.string_input_producer([data_path2], num_epochs=1)
# Define a reader and read the next record
reader = tf.TFRecordReader()
_, serialized_example = reader.read(filename_queue)
# Decode the record read by the reader
features = tf.parse_single_example(serialized_example, features=feature)
# Convert the image data from string back to the numbers
image = tf.decode_raw(features['train/image'], tf.float32)

# Cast label data into int32
label = tf.cast(features['train/label'], tf.int32)
# Reshape image data into the original shape
image = tf.reshape(image, [150,150,1])
images1,labels1 = tf.train.shuffle_batch([image, label], batch_size=10, capacity=30, num_threads=1, min_after_dequeue=10)

train_x2=images1
train_y2=labels1

input1=Input(shape=(150,150,1))
x=Conv2D(32, kernel_size=(3,3), activation='relu',padding='same')(input1)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(64, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(128, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(64, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(32, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
branch1=Flatten()(x)
model1=Model(inputs=input1,outputs=branch1)
​

input2=Input(shape=(150,150,1))
x=Conv2D(32, kernel_size=(3,3), activation='relu',padding='same')(input2)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(64, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(128, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(64, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
x=Conv2D(32, kernel_size=(3,3), activation='relu',padding='same')(x)
x=MaxPooling2D((2,2),padding='same')(x)
x=Dropout(0.25)(x)
branch2=Flatten()(x)
model2=Model(inputs=input2,outputs=branch2)
​

from keras.layers.merge import concatenate
model_list=[branch1, branch2]
merged = concatenate(model_list,axis=1)
x=Dense(10, activation='relu')(merged)
x=Dense(10, activation='relu')(x)
output= Dense(3, activation='softmax')(x)
final_model=Model(inputs=[input1,input2],outputs=output)
​

(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)

compile the model

Adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
final_model.compile(optimizer=Adam, loss='categorical_crossentropy', metrics = ['accuracy'])
​
​

train_y2],
final_model.fit([train_x1,train_x2],[train_y1,train_y2],steps_per_epoch=20,epochs=10,verbose=1)

---

can someone please guide me what i should do .....I am new to this field so i got no idea how to fix this.
thanks in addvance

[rvk007]

hi, I have a doubt. In my code,

model.fit(x=[X_train_1,X_train_2],y= Y]

the error is x=ndim is not applicable on a list, how do I rectify this???
The model is compiling fine. but fit() is giving an error.

[guofuzheng]

@fchollet Hi , I have met a problem. I want to merge two output of different networks with different weights. And the weights are computed by another network. What should I do to finish this? Thank you for your help.

[tnlin]

If you are using customize callback metric (in my case, F1-macro for multiclass classification), you may encounter the problem like this

original setting with 1 input

class Metrics(Callback):
    def on_train_begin(self, logs={}):
        self.f1s = []
        self.recalls = []
        self.precisions = []

    def on_epoch_end(self, epoch, logs={}):
        x_input = self.validation_data[0]
        predict = np.asarray(self.model.predict(x_input))
        targ = self.validation_data[1]
        
        predict = np.argmax(predict, axis=1)
        targ = np.argmax(targ, axis=1)

        _f1 = f1_score(targ, predict, average='macro')
        _recall = recall_score(targ, predict, average='macro')
        _precision = precision_score(targ, predict, average='macro')
        self.f1s.append(_f1)
        self.recalls.append(_recall)
        self.precisions.append(_precision)
        logger.info(" — val_f1: %.4f — val_precision: %.4f — val_recall %f" %(_f1, _precision, _recall))

        if self.best_f1 > _f1:
            self.wait += 1
            if self.wait >= self.patience:
                logger.info("Epoch %d: early stopping threshold" % epoch)
                self.model.stop_training = True
                self.model.set_weights(self.best_weights)
        else:
            self.best_f1 = _f1
            self.best_weights = self.model.get_weights()
            self.wait = 0

        return

modified setting with 2 input

class Metrics(Callback):
    def on_train_begin(self, logs={}):
        self.f1s = []
        self.recalls = []
        self.precisions = []

    def on_epoch_end(self, epoch, logs={}):
        x_input = [self.validation_data[0], self.validation_data[1]]
        predict = np.asarray(self.model.predict(x_input))
        targ = self.validation_data[2]
        
        predict = np.argmax(predict, axis=1)
        targ = np.argmax(targ, axis=1)

        _f1 = f1_score(targ, predict, average='macro')
        _recall = recall_score(targ, predict, average='macro')
        _precision = precision_score(targ, predict, average='macro')
        self.f1s.append(_f1)
        self.recalls.append(_recall)
        self.precisions.append(_precision)
        logger.info(" — val_f1: %.4f — val_precision: %.4f — val_recall %f" %(_f1, _precision, _recall))

        return

you can see the difference at x_input parts, I expect
self.validation_data[0] = [main_input, auxiliary_input]
but in reality, it will be
self.validation_data[0], self.validation_data[1] = main_input, auxiliary_input

Hope this could help those who have encountered the same problem.

[AloshkaD]

@fchollet how I would go about concatenating a 3 element array input with image inputs? Here is what I have

`rgb_img_input = Input(tensor=rgb_img)

d_img_input = Input(tensor=d_img)
vel_input = Input(tensor=vel)
conv_rgb_1=Conv2D(32,(4, 4), strides=(4, 4), activation='relu',input_shape=rgb_img_input.shape , data_format = "channels_last")(rgb_img_input)
conv_rgb_2=Conv2D(64,(3, 3), strides=(2, 2), activation='relu')(conv_rgb_1)
conv_d_1=Conv2D(32,(4, 4), strides=(4, 4), activation='relu',input_shape=d_img_input.shape , data_format = "channels_last")(d_img_input)
conv_d_2=Conv2D(64,(3, 3), strides=(2, 2), activation='relu')(conv_d_1)
flat_rgb=Flatten()(conv_rgb_2)
flat_d=Flatten()(conv_d_2)
flat_vel=tf.reshape((vel_input),(-1,3))
merge = concatenate([flat_d, flat_rgb,flat_vel])
hidden1 = Dense(256, activation='relu')(merge)
hidden2 = Dense(256, activation='relu')(hidden1)
predictions = Dense(nb_actions, kernel_initializer='zeros', activation='linear')(hidden2)` 

Running this gives the error

ValueError: A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 10944), (None, 10944), (1, 3)]
Concatenating the images without the velocity array works fine. I tried to reshape it with different way but nothing worked.

[zhar97]

...
# fine-tune the model history = model.fit_generator( [train_generator1, train_generator2], steps_per_epoch=nb_train_samples//32, epochs=nb_epoch, validation_data=[validation_generator1,validation_generator2], validation_steps = nb_validation_samples//32, callbacks=[reduce_lr])

but got the following error

`---------------------------------------------------------------------------
AttributeError Traceback
...

@scstu is it compulsory to use fit_generator method instead of just fit? I thought the current version already make fit scalable. I'm also using images DataIterator built with ImageDataGenerator class, but getting the following error instead

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'keras_preprocessing.image.directory_iterator.DirectoryIterator'>"}), <class 'NoneType'>

when I tried model.fit( [train_gen1, train_gen2], ... )

Does multiple inputs feature work only with numpy arrays?