Set part of the model retrainable.

[longdt219]

Hi,
I'm looking at elmo model on Tensorflow hub (https://www.tensorflow.org/hub/modules/google/elmo/1)
using the example code

elmo = hub.Module("https://tfhub.dev/google/elmo/1", trainable=True)
tokens_input = [["the", "cat", "is", "on", "the", "mat"],
                ["dogs", "are", "in", "the", "fog", ""]]
tokens_length = [6, 5]
embeddings = elmo(
    inputs={
        "tokens": tokens_input,
        "sequence_len": tokens_length
    },
    signature="tokens",
    as_dict=True)["elmo"]

They mentioned that they set trainable=True when creating the module so that the 4 scalar weights (as described in the paper) can be trained. In this setting, the module still keeps all other parameters fixed.

However, when I print set of trainable parameters from tf.trainable_variables(), I see the full list of parameters (i.e. 75m parameters). I would expect only 4 scalar weights as trainable. Could you please explain this ?

[andresusanopinto]

You are correct, sorry for the delay to update here. There was an issue on the creation of the module that declared more variables as trainable than should be. The creators have issued a new version with that fixed.

https://tfhub.dev/google/elmo/2

Changelog for Version 2
Restricted trainable variables to the 4 scalar weights as described in the paper.

Please let us know if you find more issues.

[longdt219]

Thanks for the update.