Set specific weights to non-trainable instead of the whole layer

[iliaschalkidis]

I'm trying to find a way to turn specific weights (variables) from trainable to non-trainable. For example, let's say I load a pre-trained model that includes a Dense layer and want to keep the W matrix trainable while disabling the bias term.

I see only examples of how to turn a whole layer:

import tensorflow as tf

# Make a model with 2 layers
layer1 = tf.keras.layers.Dense(3, activation="relu")
layer2 = tf.keras.layers.Dense(3, activation="sigmoid")
model = tf.keras.Sequential([tf.keras.Input(shape=(3,)), layer1, layer2])

# Freeze the first layer
layer1.trainable = False

model.summary()

Returns:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 3)                 12        
_________________________________________________________________
dense_1 (Dense)              (None, 3)                 12        
=================================================================
Total params: 24
Trainable params: 12
Non-trainable params: 12
_________________________________________________________________

What if, I want to do something like:

import tensorflow as tf

# Make a model with 2 layers
layer1 = tf.keras.layers.Dense(3, activation="relu")
layer2 = tf.keras.layers.Dense(3, activation="sigmoid")
model = tf.keras.Sequential([tf.keras.Input(shape=(3,)), layer1, layer2])

# Freeze the first layer
layer1.bias.trainable = False

model.summary()

This returns the error:

layer1.bias.trainable = False.  AttributeError: can't set attribute

[amahendrakar]

@iliaschalkidis,
Could you please take a look at this similar StackOverflow query and let us know if this is what you were looking for. Thanks!

[iliaschalkidis]

Hi @amahendrakar,

Unfortunately not. In this example, we just do not use a bias term at all. In my case, I just want to pick up a specific weight, for example, the bias term or the W matrix of a Dense layer, and turn it to no trainable, not ignore/delete it.

Thanks.

[amahendrakar]

@ymodak,
I was able to reproduce the issue with TF v2.3, TF v2.4 and TF-nightly. Please find the gist of it here. Thanks!

[ymodak]

Perhaps you can set arg use_bias=False in the layers. See gist.
Thanks!

[iliaschalkidis]

Hi @ymodak, @amahendrakar already proposed this solution and this is not the case. If we set use_bias=False, then the layer does not include a bias term. My questions is how to turn any variable, such as a bias term, to non trainable, not omit the parameter.

Thanks.

[alieskici]

It's a vital function and it's interesting that it still doesn't.

[don-tpanic]

@ymodak
Any progress on this? thanks!

[jvishnuvardhan]

@iliaschalkidis This is more of a feature (and not a bug with TF/keras)

If you still interested in this feature, open the issue as a feature in keras-team/keras repo as Keras development moved there to focus entirely on Keras. Also, mention more details about your use case where you would like to have this functionality.

Keras development moved to another repository to focus on only keras. Could you please repost this issue on keras-team/keras repo.

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[edmondja]

I almost had the same problem and I solved it this way :
conv1.non_trainable_weights.append(conv1.trainable_weights.pop(0)).

It was useful for me in order to prevent depthwise weights to train in a SeparableConv1D.