Custom Generator crash for variable input dimensions

[Cerno-b]

When I try to run a generator for a classifier that allows varying input dimensions, I get a crash when my first two batches happen to have the same input dimensions. It works fine if the two first batches have different input dimensions.

It seems that keras looks at the first two batches and if they have the same dimensions, it expects all following batches to have the same dimensions as well. If they don't, I get a crash.

This has stumped me for quite some time until I found what seemed to be the cause, and I think it could be handled in a more transparent way to help users identify the cause faster if it occurs.

I am using Keras 3.3.3

This is the error:
TypeError: 'generator' yielded an element of shape (10, 16, 32, 1) where an element of shape (None, 16, 16, 1) was expected.

The following code shows the problem:

from tensorflow.keras import Model
from tensorflow.keras.layers import Conv2D, GlobalAveragePooling2D, Dense, Input
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import Sequence

import numpy as np


class DataGenerator(Sequence):
    def __init__(self, *, show_bug, **kwargs):
        super().__init__(**kwargs)

        self.batch_size = 10

        if show_bug:
            self.batches = [
                # bug cause: first two batches have same dimensions
                np.random.rand(self.batch_size, 16, 16, 1),
                np.random.rand(self.batch_size, 16, 16, 1),
                np.random.rand(self.batch_size, 16, 32, 1),
            ]
        else:
            self.batches = [
                # first two batches have different dimensions
                np.random.rand(self.batch_size, 16, 16, 1),
                np.random.rand(self.batch_size, 16, 32, 1),
                np.random.rand(self.batch_size, 16, 16, 1),
            ]

    def __len__(self):
        return len(self.batches)

    def __getitem__(self, index):
        y = np.zeros((self.batch_size, 3))
        y[:, 0] = 1.0
        return self.batches[index], y


def run():
    inputs = Input(shape=(None, None, 1))
    layer = Conv2D(filters=16, kernel_size=(5, 5), name="conv1", activation="relu")(inputs)
    layer = GlobalAveragePooling2D(name="gapool1")(layer)
    outputs = Dense(units=3, name="dense1", activation="relu")(layer)

    model = Model(inputs=inputs, outputs=outputs)

    model.summary()

    optimizer = Adam(learning_rate=0.001)
    loss = "categorical_crossentropy"
    metrics = ['accuracy']

    # set to False to disable the bug
    datagen = DataGenerator(show_bug=True)

    model.compile(loss=loss, optimizer=optimizer, metrics=metrics)
    model.fit(datagen, epochs=300, shuffle=False)


if __name__ == '__main__':
    run()

[Cerno-b]

Some thoughts without any knowledge about the internals of Keras, so I may be way off the mark here:

I understand that due the dynamic nature of a generator, Keras has no previous knowledge about whether or not the generator will produce consistent dimensions, and Keras can probably optimize better with fixed dimensions, so it might be important to make this decision early on in training.

However, there are a few hints that I think could be used to avoid this issue:

First, a model with varying input dimensions needs to have its input defined as some version of Input(shape=(None, None, 1)). So the fit function will know that the model may receive varying input dimensions when run with a generator. So it could run in variable size mode using this information instead of using the first two batch dimensions.

It could still be desirable to run a model with variable inputs with a fixed dimension generator, so maybe there could be another argument for the fit function or a virtual function in the generator, like fixed_input_size, that is defaults to True if Keras specifically knows the sizes in advance and to False otherwise. A user would then be able to set it to true in the fixed-dimension generator case to enable optimizations.

In any case, I think it would be helpful to mention this side-effect in the error message, something like: TypeError: 'generator' yielded an element of shape (10, 16, 32, 1) where an element of shape (None, 16, 16, 1) was expected. Your model contains variable input dimensions and you use a generator. Make sure that the generator's first two batches do not have the same dimensions.

[fchollet]

Thanks for the report.

@hertschuh do you think we could do some per-batch shape validation in the data adapter in order to raise a more helpful error message? e.g.

In any case, I think it would be helpful to mention this side-effect in the error message, something like: TypeError: 'generator' yielded an element of shape (10, 16, 32, 1) where an element of shape (None, 16, 16, 1) was expected. Your model contains variable input dimensions and you use a generator. Make sure that the generator's first two batches do not have the same dimensions.

[hertschuh]

Fix is pushed. Thanks!

[Cerno-b]

Wow, that was fast. Thanks a lot!