New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix different results over three backends for ResNet50 and MobileNet #9473

Merged
merged 1 commit into from Feb 24, 2018

Conversation

Projects
None yet
5 participants
@taehoonlee
Copy link
Member

taehoonlee commented Feb 24, 2018

As discussed in #9457, there are the discrepancies in ResNet50 and MobileNet. The tests scripts are:

import numpy as np
from keras.preprocessing import image
from keras.applications.mobilenet import decode_predictions

from keras.applications.mobilenet import MobileNet
from keras.applications.mobilenet import preprocess_input as p1
from keras.applications.resnet50 import ResNet50
from keras.applications.resnet50 import preprocess_input as p2

model1 = MobileNet(weights='imagenet')
model2 = ResNet50(weights='imagenet')

img = image.load_img('cat.png', target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x1 = p1(x.copy())
x2 = p2(x.copy())

print('Predicted:', decode_predictions(model1.predict(x1), top=3)[0])
print('Predicted:', decode_predictions(model2.predict(x2), top=3)[0])

The outputs of CNTK and Theano are:

 $ KERAS_BACKEND=cntk python test_mobilenet.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.25886607), (u'n02124075', u'Egyptian_cat', 0.14433344), (u'n02123045', u'tabby', 0.06473238)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17055313), (u'n02123597', u'Siamese_cat', 0.11803201), (u'n03482405', u'hamper', 0.055440076)])

 $ KERAS_BACKEND=theano python test_mobilenet.py
Using Theano backend.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.2602814), (u'n02124075', u'Egyptian_cat', 0.14234008), (u'n02123045', u'tabby', 0.06521355)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17173238), (u'n02123597', u'Siamese_cat', 0.10258683), (u'n02130308', u'cheetah', 0.0565808)])

The problem is that TensorFlow differs as follows:

 $ KERAS_BACKEND=tensorflow python test_mobilenet.py
Using TensorFlow backend.
('Predicted:', [(u'n03482405', u'hamper', 0.21035317), (u'n02123159', u'tiger_cat', 0.113607496), (u'n04074963', u'remote_control', 0.09508652)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.116303556), (u'n02123597', u'Siamese_cat', 0.075559795), (u'n02130308', u'cheetah', 0.048775144)])

The main problem is from different padding behaviour of Conv2D(filters, kernel, padding='same', strides=(2, 2)). With explicit ZeroPadding2D, TensorFlow is able to show small numerical differences.

 $ git checkout fix_resnet_mobilenet
Switched to branch 'fix_resnet_mobilenet'
 $ KERAS_BACKEND=tensorflow python test_mobilenet.py
Using TensorFlow backend.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.2602828), (u'n02124075', u'Egyptian_cat', 0.14233977), (u'n02123045', u'tabby', 0.0652139)])
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17173262), (u'n02123597', u'Siamese_cat', 0.102586426), (u'n02130308', u'cheetah', 0.056580666)])
@fchollet
Copy link
Collaborator

fchollet left a comment

Thanks for looking into this. LGTM.

@fchollet fchollet merged commit c2a8450 into keras-team:master Feb 24, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@taehoonlee taehoonlee deleted the taehoonlee:fix_resnet_mobilenet branch Feb 25, 2018

dschwertfeger added a commit to dschwertfeger/keras that referenced this pull request Feb 26, 2018

Merge remote-tracking branch 'upstream/master' into tensorboard-callb…
…ack-embeddings-from-layer-outputs

* upstream/master: (443 commits)
  Fix `pool_2d` of Theano for backend compatibility (#9479)
  Clean up conv backend tests (#9478)
  Refactoring of ConvLSTM2D. Added ConvRNN2D and ConvLSTM2DCell. (#9094)
  Fix different results over three backends for ResNet50 and MobileNet (#9473)
  Add depthwise conv2d for Theano and CNTK (#9457)
  Fix ImageDataGenerator preprocessing_function (#9273)
  Add separable conv2d for CNTK (#9442)
  Remove word “shuffled” from comments in examples (#9453)
  Add random brightness to Image Preprocessing (Code Cleanup) (#9390)
  Only print Theano RNN dropout warning when needed.
  Add train test split to DirectoryIterator (#6152)
  Misc: Slight optimisation (#9445)
  Force update of Sequences for Windows (#9436)
  Move tests for datasets (#9439)
  Style fixes (#9441)
  Increase test coverages by excluding several lines (#9428)
  Fixing minor bug in pretrained_word_embeddings example (#9438)
  Optimizer - set_weights : check weights length (#9435)
  Add `conv_utils_test` (#9429)
  Enable `variable_input_channels` test for `InceptionV3` (#9425)
  ...
@ozabluda

This comment has been minimized.

Copy link
Contributor

ozabluda commented Mar 11, 2018

@taehoonlee, since you already understand the issue (I don't), could you, please, document the difference in behavior of Keras Conv2D(padding='same') and friends, across backends. Maybe this will answer my question why we synchronize this way (instead of, say "fixing" Theano/CNTK Conv2D and friends).

@fchollet

@taehoonlee

This comment has been minimized.

Copy link
Member

taehoonlee commented Mar 12, 2018

@ozabluda, Here is a toy example:

import numpy as np

from keras.layers import Conv2D
from keras.models import Sequential
from keras import backend as K

w = [np.arange(3*3*5*2).reshape(3, 3, 5, 2)]
if K.backend() == 'theano':
    w = [np.flip(np.flip(w[0], 0), 1)]

model = Sequential()
model.add(Conv2D(2, kernel_size=3, strides=2, input_shape=(6, 6, 5),
                 padding='SAME', use_bias=False))
model.set_weights(w)
model.predict(np.ones((1, 6, 6, 5)))

The results are:

 $ KERAS_BACKEND=theano python test.py
array([[[[1280., 1300.],
         [1770., 1800.],
         [1770., 1800.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]]]], dtype=float32)
 $ KERAS_BACKEND=cntk python test.py
array([[[[1280., 1300.],
         [1770., 1800.],
         [1770., 1800.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]],

        [[1470., 1500.],
         [1980., 2025.],
         [1980., 2025.]]]], dtype=float32)
 $ KERAS_BACKEND=tensorflow python test.py
array([[[[1980., 2025.],
         [1980., 2025.],
         [1170., 1200.]],

        [[1980., 2025.],
         [1980., 2025.],
         [1170., 1200.]],

        [[ 870.,  900.],
         [ 870.,  900.],
         [ 480.,  500.]]]], dtype=float32)

The difference comes from the difference in padding behavior when padding='same', strides=2. Here's a little more expansion:

Theano CNTK TensorFlow
odd kernel / odd input
odd kernel / even input differs
even kernel / odd input differs
even kernel / even input differs
@goravkaul

This comment has been minimized.

Copy link

goravkaul commented Jun 7, 2018

I'm getting same output for 1by1 kernel with different padding, for tensorflow backend:

conv5 = Conv2D(16,kernel_size=(1,1),strides=1,padding='same')
conv6 = Conv2D(16,kernel_size=(1,1),strides=1,padding='valid')
conv7 = Conv2D(16,kernel_size=(1,1),strides=2,padding='same')
conv8 = Conv2D(16,kernel_size=(1,1),strides=2,padding='valid')

x10=inputs
xx5 = conv5(x10)
xx6 = conv6(x10)
xx7 = conv7(x10)
xx8 = conv8(x10)

print("Input shape is:",inputs.shape)
print("kernel_size=(1,1),strides=1,padding='same'",xx5.shape)
print("kernel_size=(1,1),strides=1,padding='valid'",xx6.shape)
print("kernel_size=(1,1),strides=2,padding='same'",xx7.shape)
print("kernel_size=(1,1),strides=2,padding='valid'",xx8.shape)

Output:
('Input shape is:', TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(3)]))
("kernel_size=(1,1),strides=1,padding='same'", TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(16)]))
("kernel_size=(1,1),strides=1,padding='valid'", TensorShape([Dimension(None), Dimension(32), Dimension(32), Dimension(16)]))
("kernel_size=(1,1),strides=2,padding='same'", TensorShape([Dimension(None), Dimension(16), Dimension(16), Dimension(16)]))
("kernel_size=(1,1),strides=2,padding='valid'", TensorShape([Dimension(None), Dimension(16), Dimension(16), Dimension(16)]))

In the above results, xx5 and xx6 have same shape (xx7 and xx8 also ) even though different padding is used for convolution.

@masoodmortazavi

This comment has been minimized.

Copy link

masoodmortazavi commented Aug 30, 2018

It is really amazing to me that padding which should be an integer has been left to be interpreted from some strange set of strings (for ease of use?). When ease of use destroys clarity, it is time for some upgrade :-) .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment