Multi output network generator

[hassaan90]

model.fit support multi input/output network, but if data-set is large enough and one have to use model.fit_generator, its complicated to generate tuple for such case, is there any plan to make it more simpler like model.fit.

I have network that take one input and produce two outputs, i created separate generator for each, but i am not able to run network on this. my input should be of form x, [y1, y2].

i think i need to extend generator for such case ?

image_datagen = ImageDataGenerator(
          rotation_range=15.,
          width_shift_range=0.2,
          height_shift_range=0.2,
          rescale=1./255,
          shear_range=0.2,
          zoom_range=0.2,
          horizontal_flip=True,
          vertical_flip = True,
          fill_mode='nearest')
  
  mask_datagen = ImageDataGenerator(
          rotation_range=15.,
          width_shift_range=0.2,
          height_shift_range=0.2,
          rescale=1./255,
          shear_range=0.2,
          zoom_range=0.2,
          horizontal_flip=True,
          vertical_flip = True,
          fill_mode='nearest')
  shading_datagen = ImageDataGenerator(
          rotation_range=15.,
          width_shift_range=0.2,
          height_shift_range=0.2,
          rescale=1./255,
          shear_range=0.2,
          zoom_range=0.2,
          horizontal_flip=True,
          vertical_flip = True,
          fill_mode='nearest')
  test_datagen = ImageDataGenerator(rescale=1./255)
  seed_validation = 1
  validation_image_generator = test_datagen.flow_from_directory(
          os.path.join(gen_path,'clean/test'),
          target_size=(128, 256),class_mode=None,classes=None,
          batch_size=20,seed=seed_validation)
  validation_mask_generator = test_datagen.flow_from_directory(
          os.path.join(gen_path,'albedo/test'),
          target_size=(128, 256),class_mode=None,classes=None,
          batch_size=20,seed=seed_validation)
  validation_shading_generator = test_datagen.flow_from_directory(
          os.path.join(gen_path,'gray_shading/test'),
          target_size=(128, 256),class_mode=None,classes=None,
          batch_size=20,seed=seed_validation)
  #testDataGen = ImageDataGenerator(rescale=1./255)
  seed = 5
  #image_datagen.fit(image_datagen, augment=True, seed=seed)
  #mask_datagen.fit(mask_datagen, augment=True, seed=seed)
  
  image_generator = image_datagen.flow_from_directory(
      os.path.join(gen_path,'clean/train'),
      class_mode=None,target_size=(128,256),batch_size=20,classes=None,
      seed=seed)
  
  mask_generator = mask_datagen.flow_from_directory(
      os.path.join(gen_path,'albedo/train'),
      class_mode=None,target_size=(128,256),batch_size=20,classes = None,
      seed=seed)
  shading_generator = shading_datagen.flow_from_directory(
      os.path.join(gen_path,'gray_shading/train'),
      class_mode=None,target_size=(128,256),batch_size=20,classes = None,
      seed=seed)
  
  train_generator = itertools.izip(image_generator, mask_generator, shading_generator)

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[hassaan90]

Model is like this

[zafarali]

My experience with using fit_generator on a multi-input model required a generator to return something like: ([x1, x2], y).

My intuition leads me to believe that for a multi-output model you need to yield (x1, [y1, y2])

I would write a quick "generator joiner": (untested)

def join_generators(generators):
    while True: # keras requires all generators to be infinite
        data = [next(g) for g in generators]

        x = [d[0] for d in data ]
        y = [d[1] for d in data ]

        yield x, y

Note the assumption here is that you need to ensure that the order in which your data flows from the directory is the same for each generator

[hassaan90]

Thanks
Sorry if question sound silly
I am not able to understand input for this join_generator ? its single zipped three generators ??
and with that code y ll still lead to one input, should i write separate y1 and y2 and yield x, [y1,y2]

[hassaan90]

yes i am using same seed for all generators, and i tested that as well.

[zafarali]

Sorry if question sound silly
I am not able to understand input for this join_generator ? its single zipped three generators ??

Don't worry, I didn't include a doc string.
It's just a list of the generators. Each generator must return (x, y).
Therefore if you have:

g1 --> (x1, y1)
g2 --> (x2, y2)
join_generators(g1, g2) --> ([x1, x2], [y1, y2])

and with that code y ll still lead to one input, should i write separate y1 and y2 and yield x, [y1,y2]

This was just a skeleton of a possible implementation. For example, if your generators only return one element you could just:

def join_generators(xgenerators, ygenerator):
    while True: # keras requires all generators to be infinite
        data = [next(g) for g in xgenerators]

        yield x, next(ygenerator)

Or vary the code accordingly.

[0x00b1]

I like that trick. In my experience, writing a custom generator that yields both inputs is easier to manage than zipping two generators. It is especially evident when managing complex augmentation and sampling methods.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[Sucran]

@zafarali Thanks for that trick, this really helps me a lot a lot !

@0x00b1

Writing a custom generator that yields inputs in the right way for multiple-input or multiple-output model, is really easier to manage than zipping two generators

especially we can handle a dict or a list for return, which the fit_generator allows the x of (x, y, sample_weight) to be a list or dict (see #2568)

Hope this trick can show up in the keras documents
@fchollet

[moondra2017]

@zafarali How do we handle shuffling the dataset if we use "flow_from_directory"?

[zafarali]

I am not sure, I haven’t used the more advanced features of keras in a while and this function is unfamiliar to me. However, If it returns a generator, it should work.

…

On Wed, Jan 30, 2019 at 9:23 AM moondra2017 ***@***.***> wrote: @zafarali <https://github.com/zafarali> How do we handle shuffling the dataset if we use "flow_from_directory"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7003 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGAO_Mr9JQe89B7WP405nDMyh1rA73Auks5vIar4gaJpZM4N7KwF> .

[hassaan90]

@moondra2017 in my opinion seed parameter control this. It use same randomness. try on multiple generator with Shuffule=True

[tinalegre]

@hassaan90 might you have a github link where I could find your full model? i think, it would be a nice reference to see how to build/train/test multi-output models in Keras. My current model is unfortunately not converging when I add the second output branch and I'm not so sure, what I'm doing wrong.

[tamaraalshekhli]

@tinalegre and @hassaan90 same here I have the same problem, trying to train multi- output U-net and the code doesn't work, any progress did you resolve the problem?