Fix different results over three backends for ResNet50 and MobileNet

[taehoonlee]

As discussed in #9457, there are the discrepancies in ResNet50 and MobileNet. The tests scripts are:

import numpy as np
from keras.preprocessing import image
from keras.applications.mobilenet import decode_predictions

from keras.applications.mobilenet import MobileNet
from keras.applications.mobilenet import preprocess_input as p1
from keras.applications.resnet50 import ResNet50
from keras.applications.resnet50 import preprocess_input as p2

model1 = MobileNet(weights='imagenet')
model2 = ResNet50(weights='imagenet')

img = image.load_img('cat.png', target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x1 = p1(x.copy())
x2 = p2(x.copy())

print('Predicted:', decode_predictions(model1.predict(x1), top=3)[0])
print('Predicted:', decode_predictions(model2.predict(x2), top=3)[0])

The outputs of CNTK and Theano are:

 $ KERAS_BACKEND=cntk python test_mobilenet.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.25886607), (u'n02124075', u'Egyptian_cat', 0.14433344), (u'n02123045', u'tabby', 0.06473238)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17055313), (u'n02123597', u'Siamese_cat', 0.11803201), (u'n03482405', u'hamper', 0.055440076)])

 $ KERAS_BACKEND=theano python test_mobilenet.py
Using Theano backend.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.2602814), (u'n02124075', u'Egyptian_cat', 0.14234008), (u'n02123045', u'tabby', 0.06521355)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17173238), (u'n02123597', u'Siamese_cat', 0.10258683), (u'n02130308', u'cheetah', 0.0565808)])

The problem is that TensorFlow differs as follows:

 $ KERAS_BACKEND=tensorflow python test_mobilenet.py
Using TensorFlow backend.
('Predicted:', [(u'n03482405', u'hamper', 0.21035317), (u'n02123159', u'tiger_cat', 0.113607496), (u'n04074963', u'remote_control', 0.09508652)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.116303556), (u'n02123597', u'Siamese_cat', 0.075559795), (u'n02130308', u'cheetah', 0.048775144)])

The main problem is from different padding behaviour of Conv2D(filters, kernel, padding='same', strides=(2, 2)). With explicit ZeroPadding2D, TensorFlow is able to show small numerical differences.

 $ git checkout fix_resnet_mobilenet
Switched to branch 'fix_resnet_mobilenet'
 $ KERAS_BACKEND=tensorflow python test_mobilenet.py
Using TensorFlow backend.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.2602828), (u'n02124075', u'Egyptian_cat', 0.14233977), (u'n02123045', u'tabby', 0.0652139)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17173262), (u'n02123597', u'Siamese_cat', 0.102586426), (u'n02130308', u'cheetah', 0.056580666)])

[fchollet]

Thanks for looking into this. LGTM.

[ozabluda]

@taehoonlee, since you already understand the issue (I don't), could you, please, document the difference in behavior of Keras Conv2D(padding='same') and friends, across backends. Maybe this will answer my question why we synchronize this way (instead of, say "fixing" Theano/CNTK Conv2D and friends).

@fchollet

[taehoonlee]

@ozabluda, Here is a toy example:

import numpy as np

from keras.layers import Conv2D
from keras.models import Sequential
from keras import backend as K

w = [np.arange(3*3*5*2).reshape(3, 3, 5, 2)]
if K.backend() == 'theano':
    w = [np.flip(np.flip(w[0], 0), 1)]

model = Sequential()
model.add(Conv2D(2, kernel_size=3, strides=2, input_shape=(6, 6, 5),
                 padding='SAME', use_bias=False))
model.set_weights(w)
model.predict(np.ones((1, 6, 6, 5)))

The results are:

 $ KERAS_BACKEND=theano python test.py
array([[[[1280., 1300.],
         [1770., 1800.],
         [1770., 1800.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]]]], dtype=float32)
 $ KERAS_BACKEND=cntk python test.py
array([[[[1280., 1300.],
         [1770., 1800.],
         [1770., 1800.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]]]], dtype=float32)
 $ KERAS_BACKEND=tensorflow python test.py
array([[[[1980., 2025.],
         [1980., 2025.],
         [1170., 1200.]],

        [[1980., 2025.],
         [1980., 2025.],
         [1170., 1200.]],

        [[ 870.,  900.],
         [ 870.,  900.],
         [ 480.,  500.]]]], dtype=float32)

The difference comes from the difference in padding behavior when padding='same', strides=2. Here's a little more expansion:

	Theano	CNTK	TensorFlow
odd kernel / odd input
odd kernel / even input			differs
even kernel / odd input	differs
even kernel / even input	differs

[goravkaul]

I'm getting same output for 1by1 kernel with different padding, for tensorflow backend:

conv5 = Conv2D(16,kernel_size=(1,1),strides=1,padding='same')
conv6 = Conv2D(16,kernel_size=(1,1),strides=1,padding='valid')
conv7 = Conv2D(16,kernel_size=(1,1),strides=2,padding='same')
conv8 = Conv2D(16,kernel_size=(1,1),strides=2,padding='valid')

x10=inputs
xx5 = conv5(x10)
xx6 = conv6(x10)
xx7 = conv7(x10)
xx8 = conv8(x10)

print("Input shape is:",inputs.shape)
print("kernel_size=(1,1),strides=1,padding='same'",xx5.shape)
print("kernel_size=(1,1),strides=1,padding='valid'",xx6.shape)
print("kernel_size=(1,1),strides=2,padding='same'",xx7.shape)
print("kernel_size=(1,1),strides=2,padding='valid'",xx8.shape)

Output:
('Input shape is:', TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(3)]))
("kernel_size=(1,1),strides=1,padding='same'", TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(16)]))
("kernel_size=(1,1),strides=1,padding='valid'", TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(16)]))
("kernel_size=(1,1),strides=2,padding='same'", TensorShape([Dimension(None), Dimension(16), Dimension(16), Dimension(16)]))
("kernel_size=(1,1),strides=2,padding='valid'", TensorShape([Dimension(None), Dimension(16), Dimension(16), Dimension(16)]))

In the above results, xx5 and xx6 have same shape (xx7 and xx8 also ) even though different padding is used for convolution.

[masoodmortazavi]

It is really amazing to me that padding which should be an integer has been left to be interpreted from some strange set of strings (for ease of use?). When ease of use destroys clarity, it is time for some upgrade :-) .

[polarCatZhao]

Hi! Is/Has anybody working/worked on making Conv2D(padding='same') consistent for the Tensorflow backend? If no, can I help with that issue? I'm a first time contributor, so please let me know if I have any misunderstanding. @fchollet