Model weights are not de-duplicated on save

[mattdangerw]

When a model shares a single variable across many layers, model.weights is deduped. Only one weight will be returned no matter how many times it is referenced.

However when a model is saved, every layer that has a ref to variable will save their own copy. Weights are not deduped on save.

This can lead to a funny flow where things appear to be working, but the saved weights are actually duplicated on disk, and a single variable is assigned multiple times during load. This colab shows the issue ->
https://colab.research.google.com/gist/mattdangerw/2fabb09d53bd19cb6407d219e34155ab/weight-sharing-weirdness.ipynb

[mattdangerw]

This seems like a bug, and is at very least confusing. Unsure what the fix should be, particularly with compat concerns in mind.

[fchollet]

The level at which sharing is supported is at the layer instance level. You may reuse layer instances multiple times. However you are not supposed to share a variable instance across multiple independent layer instances. Each variable must be owned by exactly one layer.

What happens if you try to share a layer instance instead?

[mattdangerw]

Doing this at the layer level does look like it works in keras-core, thought no fully in tf.keras. Sounds like overall this is "works as intended."

I do worry that there are not guardrails for this though. It's confusing that shared weights "just work" for training, with proper deduplication, and only start misbehaving on save.

Can we just warn if we see duplicate weights on a save call? Is there ever a valid reason for it?

[fchollet]

Can we just warn if we see duplicate weights on a save call

How would we detect duplicate weights?

[abuelnasr0]

I have some comments for this issue. There is some cases where the layers can have only one shared weight but each layer has its own other weights. An example of that is TransformerXL model where each decoder layer has its own weights but the relative position bias can be shared between them.
I think saving duplicate weights can be avoided by checking the id of the weight before you save it. if a weight with the same id has been saved before we don't save it. I have implemented it locally and I think it works fine.
may be there is something I am missing about this, but this is a solution that might help.

[innat]

Hi @mattdangerw
I've just tried to run the above colab gist.

Considering that scenarios , have you encouontered any impact in model performance by doing model.save and model.save_weights? In my case, reloading the saved model with keras.models.load_model API, the memory gets fill up much higher than manually create and load weight with .load_weights API. Could it be the reason of weights not being deduped in model saving time (model.save)?

[innat]

might be related case
keras-team/tf-keras#139