Variable-size image to convolutional layer

[shodutta92]

Hi, I'm trying to run variable-sized images through convolutional layers but I'm getting an error about the row sizes not matching. From what i understand in this post, I should be able to use a snippet like

from keras.models import Sequential
from keras.layers import Convolution2D
import numpy as np

m = Sequential()
m.add(Convolution2D(8, 3, 3, input_shape=(1, 10, 10)))
m.compile(loss="mae", optimizer="sgd")
c = m.predict(np.random.rand(1, 1, 10, 10))
c = m.predict(np.random.rand(1, 1, 20, 20))

However, the second predict statement throws an error:

ValueError: The hardcoded shape for the number of rows in the image (10) isn't the run time shape (20).

Is there some way to use this in an input-shape independent fashion?

[fchollet]

You should use:

input_shape=(1, None, None)

None in a shape denotes a variable dimension. Note that not all layers will work with such variable dimensions, since some layers require shape information (such as Flatten).

[slegall56]

Ok but if I have a flatten layer in my network such as :

input = Input(shape=(None, None,3))
x = Convolution2D(10, 3, 3, border_mode='valid', init = 'he_normal')(input)
x = PReLU()(x)
x = MaxPooling2D(pool_size=(2,2))(x)
x = Convolution2D(16, 3, 3, border_mode='valid', init = 'he_normal')(x)
x = PReLU()(x)
x = Convolution2D(32, 3, 3, border_mode='valid', init = 'he_normal')(x)
x = PReLU()(x)
x = Flatten()(x)
x = Dense(2)(x)
x = Activation('softmax', name='activatioon)(x)
		
model = Model(input = input, output= x)

How can I deal with it ?

@slegall56 You should probably use GlobalAveragePooling. It is essentially an average pooling with adaptive size, and it converts different-sized convolutional feature maps into a constant size 1D vector.

[ybdesire]

How Keras internel actually handle the variable-size input, just add zero padding ?

[KeithWM]

@ybdesire The size of the weights in a convolutional layer does not depend on the size of the input image, only on the number of channels in, channels out and the size of the kernel. So there's no need to use zero padding for the weights.

For the memory used to store a batch of input images I would presume no padding is used, but images are packed in memory. @moi90 is right: all images in one batch need to be of the same size.

[moi90]

Model.train_on_batch receives a Numpy array as input, so at least in every batch the images have to have the same size. Its up to you, how you achive this. ImageDataGenerator.flow_from_directory has a parameter called target_size to resize all images of the whole dataset to a common size. This contradicts what @KeithWM said.

[aidamian]

I have a similar problem with Keras (tf backend) and I can't seem to find a solution. Wonder if I am missing something.
Basically my network architecture is defined by:

  in_layer = Input(shape=(None, None,nr_channels))
  x = Conv2D(16,(4,4), activation = 'elu')(in_layer)  # single stride 4x4 filter for 16 maps
  x = Conv2D(32,(4,4), activation = 'elu')(x)         # single stride 4x4 filter for 32 maps
  x = Dropout(0.5)(x)
  x = Conv2D(64,(4,4), activation = 'elu')(x)         # single stride 4x4 filter for 64 maps
  x = Dropout(0.5)(x)
  x = Conv2D(128, (1,1))(x)                           # finally 128 maps for global average-pool
  x = GlobalAveragePooling2D()(x)                     # pseudo-dense 128 layer
  output_layer = Dense(10, activation = "softmax")(x) # softmax output
  model = Model(inputs = input_layer, outputs=output_layer)
  model.compile(optimizer = "adam", loss = "categorical_crossentropy",
                metrics=["accuracy"])

Now I train it with 60k training mnist without any kind of modifications. Then I evaluate it on 10k test mnists (also 20x28) and I receive a 93% after couple of epochs.
However, when I pass a larger image: say 100x75 where copy in a random place 1 mnist test image and I do predict(test_image_100x75) I receive something very strange:

  Label: 3
  Prediction: [[ 0.  0.998  0.  0.  0.  0.  0.  0.  0.  0.]]
  y_test:     [ 0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]

  Label: 7
  Prediction: [[ 0.  0.995  0.  0.  0.  0.  0.  0.  0.  0.]]
  y_test:     [ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]

So, basically my model predicts label "1" for any variable size image I give. There are no other operations on the variable size test-set other than 1 random-position copy (all var sized images are checked with matshow as in this link example of var sized test image

[aidamian]

Well, I am ashamed to admit, but in my previous post I used global average pooling that basically destroyed the small-patch image information (small patch on image info in a big zero-ed image..). Obvious enough I switched to GlobalMaxPooling2D and everything worked perfectly.
So above code becomes:

  in_layer = Input(shape=(None, None,nr_channels))
  x = Conv2D(16,(4,4), activation = 'elu')(in_layer)  # single stride 4x4 filter for 16 maps
  x = Conv2D(32,(4,4), activation = 'elu')(x)         # single stride 4x4 filter for 32 maps
  x = Dropout(0.5)(x)
  x = Conv2D(64,(4,4), activation = 'elu')(x)         # single stride 4x4 filter for 64 maps
  x = Dropout(0.5)(x)
  x = Conv2D(128, (1,1))(x)                           # finally 128 maps for global average-pool
  x = GlobalMaxPooling2D()(x)                     # pseudo-dense 128 layer
  output_layer = Dense(10, activation = "softmax")(x) # softmax output
  model = Model(inputs = input_layer, outputs=output_layer)
  model.compile(optimizer = "adam", loss = "categorical_crossentropy",
                metrics=["accuracy"])

Above code works perfectly, the issue should be closed.

[shay86]

Hi,
I read and tried to implement your model on my data. Doing as you say customize my data to have same size in each (patch) inputed to my model. However, I discover that your data Lable has fixed size (10) as you Dense layer shows, but for me my input size are same as output size both are equal but varied from batch to batch. I tried to omit the Dense layer but even though the GlobalMaxPooling layer give me same error
{ValueError: Error when checking target: expected dense_1 to have shape (1, 20) but got array with shape (1, 21)}
when it starts the second batch because of the different output size. Any suggestion for this problem???? thanks

[aidamian]

So, the whole post is about using variable size images as input for a conv net by using GlobalMaxPooling just before your dense block or just final softmax classifier and "matching" the filters in last conv layer with number of dense units in first dense layer (matching does not mean equal number, it can be any number of filters and certainly the bigger the better - so the global pooling will transform each filter in a actual unit input for first dense layer in classification block). This is not about the variable size labels - labels are ... labels, and should you use softmax then labels must be one hot encoded by number of classes. Maybe I understood your question incorrectly and/or maybe you should post your code and I will gladly help.