How does ImageDataGenerator work with Merged model?

[ShunyuanZ]

So the problem is that, my validation set is too large and can't fit in memory. Then Following issue #2702, I tried to do batch on validation set with ImageDataGenerator and datagen.flow(X,y). However here comes the tricky part:
My model is a merged model, combining two sequential models. With left branch dealing with 3 channel RGB images and right branch a vector representing some text information. So the input in my CNN is {image, text}, and the output is {label}. I guess ImageDataGenerator only deals with images, so it will be problematic if multiple sources (and different types) of X (inputs) are passed. The following is what I did:

# merge two branches
from keras.layers import Merge
left_branch=model_image
right_branch=model_text
model=Sequential()
model.add(Merge([left_branch, right_branch], mode='concat'))
model.compile(loss='mean_squared_error', optimizer='adadelta',metrics=['accuracy'])

# define data generators
from keras.preprocessing.image import ImageDataGenerator
datagen_train= ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2)
datagen_validation= ImageDataGenerator(rescale=1./255)

# model fit use fit_generator

model.fit_generator(
datagen_train.flow([image_train,text_train],label_train, batch_size=20),
validation_data = datagen_validation.flow([image_validation,text_validation],label_validation, batch_size=20),
                callbacks=[TensorBoard(log_dir=mylogpath,histogram_freq=1,write_graph=False)],
                nb_epoch=60,
                shuffle=True,
                verbose=1)

Then I got the following error:

Traceback (most recent call last):
  File "mycnn.py", line 187, in <module>
    model.fit_generator(datagen.flow([image_train,text_train], label_train,batch_size=20),
  File "/usr/local/lib/python2.7/site-packages/keras/preprocessing/image.py", line 261, in flow
    save_to_dir=save_to_dir, save_prefix=save_prefix, save_format=save_format)
  File "/usr/local/lib/python2.7/site-packages/keras/preprocessing/image.py", line 454, in __init__
    'Found: X.shape = %s, y.shape = %s' % (np.asarray(X).shape, np.asarray(y).shape))
  File "/usr/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: could not broadcast input array from shape (40000,3,224,224) into shape (40000)

Where 40000 is the training-set size, (3,224,224) represent my RGB 224x224 images.

If I just ignored validation set, and do the following (disable validation in model.fit), then it worked:

model.fit([image_train,text_train],label_train,
                callbacks=[TensorBoard(log_dir=mylogpath,histogram_freq=1,write_graph=False)],
                batch_size=20,
                nb_epoch=60,
                shuffle=True,
                verbose=1)

This way, the images and texts are successfully passed through the two branches, however, I wound't be able to record and monitor validation loss in the training process....

Could someone give a suggestion? I'd like to know how to batch on validate set, or a way of making ImageDataGenerator working with Merged model. Thank you!!!

[xxxzhi]

how about conbime genrator?
in https://keras.io/preprocessing/image/, the last example.

[xxxzhi]

you can define youself Iterator like this:

from keras.preprocessing.image import Iterator

__author__ = 'houzhi'


class MergeIterator(Iterator):
    """
    iterator for merge model
    """

    def __init__(self, iter_list, N, batch_size=32, shuffle=False, seed=None, ):
        """

        :param iter_list: all iterator must have same shape and batch_size.
        :param N:
        :param batch_size:
        :param shuffle:
        :param seed:
        """
        self.iter_list = iter_list
        last_item = None
        for item in iter_list:
            if last_item is None:
                last_item = item
                continue
            if item.N != last_item.N or item.batch_size != last_item.batch_size:
                raise Exception("N and batch_size in each iterator must same!!!")
        super(MergeIterator, self).__init__(N, batch_size, shuffle, seed)

    def next(self):
        batch_x = None
        batch_y = None
        if len(self.iter_list) == 1:
            batch_x, batch_y = self.iter_list[0].next()
        else:
            batch_x = []
            for item in self.iter_list:
                temp_x, temp_y = item.next()
                if batch_y is None:
                    batch_y = temp_y
                batch_x.append(temp_x)

        return batch_x, batch_y

and use:

generator = MergeIterator([datagen1.flow(),datagen2.flow()], N = )