Vastly different output between tensorflow hub and keras for EfficientNetB0 due to possibly missing normalization

[Zahlii]

I am attempting to migrate an existing Efficientnet model from keras.applications to the respective hub. However, I see vastly different behaviors in training and final performance, and thus I am trying to find the root cause.

Even with the correct preprocessing ([0-255] for keras.applications, [0-1] for tf hub), and having checked that the weights are exactly the same between the loaded feature extractors, I get different results.

However, as shown below, the keras.applications approach actually does a Rescaling(1/255.0), followed by a Normalization() layer before passing the inputs to the rest of the layers, where tf hub (as far as I understand) only receives the already rescaled outputs, so I am wondering if there might be a discrepancy due to different ways of normalizations.

import collections

import tensorflow as tf
import tensorflow_hub as tfh

img2 = tf.io.read_file("tests/test2.jpg")
img2 = tf.image.decode_jpeg(img2)
img2 = tf.image.convert_image_dtype(img2, tf.float32)
img2 = tf.image.resize(img2, (224, 224))
print(img2.numpy().min(), img2.numpy().max(), img2.numpy().mean())
# 0.0 1.0 0.14994633

img2 = tf.expand_dims(img2, axis=0)

model = tf.keras.applications.efficientnet.EfficientNetB0(include_top=False, input_shape=(224, 224, 3), pooling="avg")
model.trainable = False

# Note: each Keras Application expects a specific kind of input preprocessing.
# For EfficientNet, input preprocessing is included as part of the model (as a Rescaling layer),
# and thus tf.keras.applications.efficientnet.preprocess_input is actually a pass-through function.
# EfficientNet models expect their inputs to be float tensors of pixels with values in the [0-255] range.

print(model(img2 * 255.0, training=False))
# [[-0.1674832   0.06747124 -0.00998851 ... -0.09722453 -0.05333998
#   -0.09630407]], shape=(1, 1280), dtype=float32)

inp = tf.keras.layers.Input(shape=(224, 224, 3))
layer = tfh.KerasLayer("https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1", trainable=False)(inp)

model_hub = tf.keras.models.Model(inputs=inp, outputs=layer)
print(model_hub(img2, training=False))
# tf.Tensor(
# [[-0.1534781  -0.00044355  0.02527641 ...  0.03095446 -0.07474983
#   -0.12997748]], shape=(1, 1280), dtype=float32)


weights = collections.defaultdict(lambda: collections.defaultdict(dict))

for w in model.non_trainable_weights:
    m = w.numpy().mean()
    weights[tuple(w.shape)][m][f"{w.name}_KERAS"] = w.numpy()

for w in model_hub.non_trainable_weights:
    m = w.numpy().mean()
    weights[tuple(w.shape)][m][f"{w.name}_HUB"] = w.numpy()


for s, data in weights.items():
    for mn, data2 in data.items():
        n = len(data2)
        has_hub = any("HUB" in name for name in data2.keys())
        has_keras = any("KERAS" in name for name in data2.keys())
        if n % 2 != 0 or not has_hub or not has_keras:
            print(s, mn, data2)

# (3,) 0.449 {'normalization/mean:0_KERAS': array([0.485, 0.456, 0.406], dtype=float32)}
# (3,) 0.226 {'normalization/variance:0_KERAS': array([0.229, 0.224, 0.225], dtype=float32)}
# () 0.0 {'normalization/count:0_KERAS': 0}

[UsharaniPagadala]

@Zahlii
Could you please refer the similar issues #42506 and 60251715 and let us know if this helps.Thanks

[Zahlii]

@UsharaniPagadala as you can see already in the code, I stumbled acorss both of these issues previously. I am normalizing accordingly (range 0 - 255 for keras, 0 - 1 for hub); and I am not even dealing with the classification variant here, just the feature vectors (which in both cases I assume return values without activation).

[alenarepina]

Please follow the tutorial https://www.tensorflow.org/hub/tutorials/tf2_image_retraining and let us know if you still see quality issues.

[Zahlii]

Copying the tutorial code leads to exactly the same (erroneous) outputs as in my example; see code snippet below.

The reason WHY I opened this ticket was because I saw a drop in performance (98% to roughly 93%) when switching from keras to tfhub (while following exactly the tutorial posted), while keeping all other hyperparameters the same (learning rate, dropout, train/test splits, etc). This, for me, indicates that there is some kind of underlying issue with the outputs provided by the (unrefined) efficient net layer on hub, hence my attempt at debugging if this is a reason of different weights, or some other problems.

import tensorflow as tf
import tensorflow_hub as hub

print("TF version:", tf.__version__)
print("Hub version:", hub.__version__)
print("GPU is", "available" if tf.config.list_physical_devices('GPU') else "NOT AVAILABLE")

model_name = "efficientnet_b0"

model_handle_map = {
  "efficientnet_b0": "https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1",
}

model_image_size_map = {
  "efficientnet_b0": 224,
}

model_handle = model_handle_map.get(model_name)
pixels = model_image_size_map.get(model_name, 224)

print(f"Selected model: {model_name} : {model_handle}")

IMAGE_SIZE = (pixels, pixels)
print(f"Input size {IMAGE_SIZE}")

BATCH_SIZE = 16


do_fine_tuning = False
print("Building model with", model_handle)
model = tf.keras.Sequential([
    # Explicitly define the input shape so the model can be properly
    # loaded by the TFLiteConverter
    tf.keras.layers.InputLayer(input_shape=IMAGE_SIZE + (3,)),
    hub.KerasLayer(model_handle, trainable=do_fine_tuning),
])
model.build((None,)+IMAGE_SIZE+(3,))
model.summary()


img2 = tf.io.read_file("tests/test2.jpg")
img2 = tf.image.decode_jpeg(img2)
img2 = tf.image.convert_image_dtype(img2, tf.float32)
img2 = tf.image.resize(img2, (224, 224))
img2 = tf.expand_dims(img2, axis=0)

print(img2.numpy().min(), img2.numpy().max(), img2.numpy().mean())
print(model.predict(img2))
# [[-0.1534781  -0.00044355  0.02527641 ...  0.03095446 -0.07474983
#  -0.12997748]]

[maringeo]

Hi @Zahlii, the preprocessing for the TF Hub model is done as tf.data transform and it's hidden by default: if you go to the Setup the flowers dataset section of the Colab and click on Toggle code you will see the transforms.

Most of the transforms are only applied during training in order to generate more training data by augmenting the input images. But tf.keras.layers.Rescaling(1. / 255) is applied during inference too.

I'm not sure what dataset and metric you refer to in I saw a drop in performance (98% to roughly 93%) but adding tf.keras.layers.Rescaling(1. / 255) to the TF Hub model will likely improve the score (tf.keras.applications.efficientnet.EfficientNetB0 also uses this layer).

I can't repro the exact behavior since I am not certain what dataset and metric are used but could you try adding tf.keras.layers.Rescaling(1. / 255) and let us know if it helps?

[Zahlii]

Hi @maringeo , as I already stated above I am aware that the rescaling needs to be done prior to feeding the image to tfhub as per the image conventions. Also, rescaling will only be required when the input is in int format, i.e. 0...255. When using the tf.convert_image_dtype approach, this rescaling is already applied, hence adding a Rescaling() layer prior to the tfhub layer actually achieves nothing, because data will be completely off scale.

The code mentioned above is directly based on https://www.tensorflow.org/hub/common_signatures/images#input .

I already checked this by confirming the input data is correctly converted into the [0...1] range just prior to calling model.predict():

print(img2.numpy().min(), img2.numpy().max(), img2.numpy().mean())
# 0.0 1.0 0.14994633

To confirm, the following modification to above produces the following values, which are WAY further away from the keras applications values than even the hub implementation.

model = tf.keras.Sequential([
    # Explicitly define the input shape so the model can be properly
    # loaded by the TFLiteConverter
    tf.keras.layers.InputLayer(input_shape=IMAGE_SIZE + (3,)),
    tf.keras.layers.experimental.preprocessing.Rescaling(1/255.0),
    hub.KerasLayer(model_handle, trainable=do_fine_tuning),
])
...
model.predict(...)
# [[-0.06889984 -0.09886284 -0.13236746 ... -0.19460136 -0.04948389
#   0.04994771]]

It seems to me that the keras applications might do some kind of channel-wise normalization, which potentially the hub version is not doing...