MoviNet: Can not assign pretrained weighted because of the difference between pretrained and defined model

[t2kien]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ ]x I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [ x] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [x ] I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/official/projects/movinet

2. Describe the bug

It seems that the when using pretrained movinet model (base version), the layer name of pretrained model and defined model is different. It show "Key Error" error when assign value of model.

3. Steps to reproduce

Steps to reproduce the behavior.
I declare a new movinet with

backbone = movinet.Movinet(
model_id=model_id)
model = movinet_model.MovinetClassifier(
backbone=backbone,
num_classes=600)
model.build([batch_size, num_frames, resolution, resolution, 3])

Then, download pretrained model from
movinet_hub_url = f'https://tfhub.dev/tensorflow/movinet/{model_id}/base/kinetics-600/classification/1'
movinet_hub_model = hub.KerasLayer(movinet_hub_url, trainable=True)

and assign pretrained weighted to my model

pretrained_weights = {w.name: w for w in movinet_hub_model.weights}
model_weights = {w.name: w for w in model.weights}
for name in pretrained_weights:
model_weights[name].assign(pretrained_weights[name])

It shows error: "KeyError: 'b0/l0/bneck/expansion/conv2d/conv2d/kernel:0'"

4. Expected behavior

5. Additional context

I fixed it temporarily by renaming keys of defined model as follows: " base -> b" and "layer ->l"

6. System information

• OS Platform and Distribution: Linux Ubuntu 20.04
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 2.7
• Python version: 3.8

[ItsCRC]

I am facing the same issue. @t2kien Can you tell more about how to rename keys from "base->b" and "layer->l"?

[jvishnuvardhan]

@t2kien @ItsCRC Can you please share simple standalone code (py/colab/jupyter notebook) to reproduce the issue? Thanks!

[ItsCRC]

model_id = "a1"
backbone = movinet.Movinet(
model_id=model_id)
model = movinet_model.MovinetClassifier(
backbone=backbone,
num_classes=600)
model.build([batch_size, num_frames, resolution, resolution, 3])

movinet_hub_url = f'https://tfhub.dev/tensorflow/movinet/{model_id}/base/kinetics-600/classification/1'
movinet_hub_model = hub.KerasLayer(movinet_hub_url, trainable=True)

# Assigning pretrained weighted to my model

pretrained_weights = {w.name: w for w in movinet_hub_model.weights}
model_weights = {w.name: w for w in model.weights}
for name in pretrained_weights:
model_weights[name].assign(pretrained_weights[name]) #Error

It shows error: "KeyError: 'b0/l0/bneck/expansion/conv2d/conv2d/kernel:0'"

[Hyperparticle]

@t2kien @ItsCRC I see you are using the v1 model version. Can you try to use the current (v3) version which has the updated weight names?

So for example you should do

https://tfhub.dev/tensorflow/movinet/a0/base/kinetics-600/classification/3

for a0 instead of

https://tfhub.dev/tensorflow/movinet/a0/base/kinetics-600/classification/1

[t2kien]

I am facing the same issue. @t2kien Can you tell more about how to rename keys from "base->b" and "layer->l"?

Actually, I just rename the keys of defined model's weight dictionary as below and it works

   pretrained_weights = {w.name: w for w in movinet_hub_model.weights}
   model_weights = {w.name: w for w in model.weights}
   for name in model_weights:
       replaced_name = name.replace("block","b")
       replaced_name = replaced_name.replace("_layer","/l")
       model_weights[name].assign(pretrained_weights[replaced_name])

@Hyperparticle : i am going to try with different versions again. I remember that I've tried with both v1 and v3 but no luck. I've also tested with stream version of both predefined and defined model, and they are different.
@jvishnuvardhan I just follow the tutorial in the movinet_tutorial.ipynb file. It is quite strange

[t2kien]

@Hyperparticle: I've tested again with model v3 and dumped both pretrained model and declared model as below:

        backbone = movinet.Movinet(model_id=model_id, causal=True)
        model = movinet_model.MovinetClassifier(
            backbone=backbone,
            num_classes=600)
        model.build([batch_size, temporal_slice, resolution, resolution, 3])
        movinet_hub_url = f'https://tfhub.dev/tensorflow/movinet/{model_id}/{model_type}/kinetics-600/classification/3'
        movinet_hub_model = hub.KerasLayer(movinet_hub_url, trainable=True)
        pretrained_weights = {w.name: w for w in movinet_hub_model.weights}
        model_weights = {w.name: w for w in model.weights}

A dump of pretrained model's weight is displayed:

print(pretrained_weights)

dict_keys(['stem/stem/conv3d/kernel:0', 'stem/stem/bn/gamma:0', 'stem/stem/bn/beta:0', 'b0/l0/bneck/expansion/conv3d/kernel:0', 'b0/l0/bneck/expansion/bn/gamma:0', 'b0/l0/bneck/expansion/bn/beta:0', 
...])

and declared model's weight dumps:

print(model_weights)

dict_keys(['stem/stem/conv3d/kernel:0', 'stem/stem/bn/gamma:0', 'stem/stem/bn/beta:0', 'stem/stem/bn/moving_mean:0', 'stem/stem/bn/moving_variance:0', 'block0_layer0/bneck/expansion/conv3d/kernel:0', 'block0_layer0/bneck/expansion/bn/gamma:0', ....])```

Finally, when assigning model weights, I have to rename it to match the other

    for name in model_weights:
        replaced_name = name.replace("block","b")
        replaced_name = replaced_name.replace("_layer","/l")
        model_weights[name].assign(pretrained_weights[replaced_name])

[Hyperparticle]

I see, we might have renamed the weights after we exported the TF Hub modules. I will make a note and make sure to update the weight names once we release the next version (v4). For now you can manually rename the weights "b" -> "block", "l" -> "layer".

[ItsCRC]

@t2kien thanks, it worked. However, it needs to be fixed as it does not work on any version @Hyperparticle.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[ItsCRC]

Any tutorial or guide for fine-tuning a stream model? I actually tried with movienet as backbone and movinet classifier with my own number of classes. It resulted in error: layer movienet-classifier-11 expects 125 input(s), but it received input tensors. ( following READ.MD , I changed params for movienet.Movinet(... )

[feckpown]

This issue has still not been fixed.

Pretty much all examples fail. The code and the pre-trained models provided do not match.