Model conversion weights size mismatch

[dbouget]

Purpose: converting a trained model's precision from float16 to float32.

Issue: the set_weights method indicates a size mismatch without recreating the model by wrapping it inside the GAModelWrapper, converts fine when wrapped.

[andreped]

I was unable to reproduce this issue. See gist.

I see that you used set_weights(), but I believe you want to use load_weights(), no?

---

EDIT: However, I get a warning "WARNING:tensorflow:Inconsistent references when loading the checkpoint into this object graph (...)". It might be related to what you are observing, but it does not result in an error.

[dbouget]

I have tried with load_weights() directly on the non-wrapped model, which does not crash and produces a new model. But I am getting plenty of warnings (most likely for all variables linked to the GAModelWrapper).

Some example:
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-40.bias WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use status.expect_partial(). See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function. WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).accum_step_counter

I guess it's fine for running inference, but should be fully wrapped to resume training.

---

Working lines, whereby the full_model has been created including the GAModelWrapper component.
mixed_model = load_model(mixed_model_filepath, compile=False)
full_model.set_weights(mixed_model.get_weights())

[andreped]

@dbouget As you found a fix, I will close this issue.

If these warning results in any breaking issues, please, let me know and I will re-open the issue again.