Resnet18 and Resnet50 has different rescaling/preprocessing implementation?

[kechan]

I noted Resnet18 has this line of code, which rescale pixel value to be [0, 1], (or [-1, 1]) don't remember which but close enough.

x = imagenet_utils.preprocess_input(
        x, data_format=data_format, mode=preproc_mode
    )

But rescaling (1/255) is probably not done for resnet50. I instantiated it and use .summary() and get_layer().get_summary() and don't see any layers.Rescale(...), while EfficientNet has this.

Whats more confusing is that in the unsupervised_hello_world notebook (https://colab.research.google.com/github/tensorflow/similarity/blob/master/examples/unsupervised_hello_world.ipynb), a comment:

"This augmenter returns values between [0, 1], but our ResNet backbone model expects the values to be between [0, 255] so we also scale the final output values in the augmentation function."

If what i observed is true, it is all well with resnet18 but not quite for resnet50.

In fact, I noticed there maybe no rescaling inside resnet50 when i tried similarity training swapping out effnet and noticed a drop in accuracy. After adding in layers.Rescale, it regains the accuracy difference.

I also understand ResNet50 rescaling may not be done in keras original implementation, before tf2.0 provided rescaling layers. So I am not sure if this is in keeping with convention. But I suspect you could have used imagenet_utils.preprocess_input(...) to do the same for resnet50? Forgetting to do 1/255. is a common mistake (at least for me) whenever a new project got started. In general, I have seen ppl done it either in tf.data.Dataset, or built it inside the model.

So I am not sure if the implementations are as intended here. I also wish for a boolean model parameter to decide to perform rescaling or not inside the model.

[owenvallis]

So this is a little tricky. The ResNet50 in TF provides the input preprocessing as a separate function (see line 506), so I left it the same way for our wrapper to be consistent with the TF application module. However, the input preprocessing for EfficientNet is included directly in the model (see line 321).

I think adding preprocessing to the ResNet50 architecture is a good call, as that makes the two ResNet architectures consistent, and I can add a boolean to disable the preprocessing. However, I don't think we can support the disable flag for the EfficientNet as the preprocessing is inlined directly in the model.

wdyt?

[kechan]

Agreed. I do realize there's some conflicting goals and need for consistency with prior ways of doing things.
I think it will be great if you can add that boolean to include/exclude the rescale processing (presumably using the more modern layers.Rescale(...) built into the model arch, just like effnet). It will be nice to keep code at minimal complexity if you can just swap in and out among all effnet*** and resnet*** families.

Aside, I notice latest tutorials from keras on transfer learning has promoted the use of tf 2.0 layers to do this pre-processing, instead of doing so in tf dataset pipeline. In particular, one from @fchollet himself used 1/255 such that pixel value is [0, 1] (i need to double check, so don't quote me on this one). However, if i remember, imagenet weight was trained with value [-1, 1]. Although I found such minor difference (if real) doesn't matter much in most fine-tuning and transfer learning.

[Aditya-Jha2002]

Hey, can I take up this issue.
@owenvallis

[owenvallis]

Hi thanks for offering to take up this issue, but I can take this one. We're actually thinking of moving the Resnet18 model over to kerasCV.

[kechan]

@owenvallis Thanks for the headups.

I also noticed there may be inconsistency in the weight init between these 2 models. ResNet18Sim has no imagenet weight init (or any weights), while ResNet50Sim has 'imagenet' as default. I am curious about any papers that actually do use init with imagenet while doing self-supervised pre-training. I suspect ResNet50Sim was coded in a mindset about supervised metric learning (not contrastive ss).

This is just something i noticed, but probably have not much opinion on what should be. Maybe ResNet18 should have an option to init with imagenet weights?

[owenvallis]

Good point. I was looking around but I'm not sure what the canonical source is for the ResNet18 ImageNet weights. I see that TF hosts the weights for the ResNet models under the applications module, but I didn't see anything for the smaller ResNets. Once I find a source for the weights, I can add the support though.

[kechan]

@owenvallis Actually, i m curious and just visited

https://pypi.org/project/kerascv/

And right there in their first example:

from kerascv.model_provider import get_model as kecv_get_model
import numpy as np

net = kecv_get_model("resnet18", pretrained=True)
x = np.zeros((1, 3, 224, 224), np.float32)
y = net.predict(x)

So maybe we can consider that one the “canonical source”, although i have seen >1 other GitHub by individual (not corp) that offer resnet family and weights.

When i get the chance, i will try to swap in some of their models to use in similarity/contrastive training, and check them out, although the project is very new, probably untested in the wild.

[owenvallis]

@kechan, I think that's actually another package with the same name. I was referring to this repo. However, the package you shared may be a good source for the ResNet18 weights and possibly the models as well.

[kechan]

@owenvallis Oh I see. Guess the best bet is to git clone that repo directly and not use any pip install. Actually, @fchollet informed me that kerasCV is likely stable (1.0) only by late 2022. I think I will still give it a try, giving it extra cautious cross check.

[owenvallis]

@kechan I finally got back to this and cleaned up the architecture modules and added test coverage. As part of that, I updated the unsupervised notebook and moved the imagenet_utils preproc into its own function in the notebook. I realized we weren't applying the scaling to the EvalCallback when using the ResNet50 and this would lead to broken match acc metrics. I think this should all work much cleaner now, but let me know if you run into any issues.