[TF 2.0] Provide a tool to convert checkpoints for optimizers, from TF 1.x to TF 2.0.

[dynamicwebpaige]

Checkpoints will break with RNNs and shared embedding columns, which is unfortunate but acceptable. For optimizers, though, checkpoint breaking is a bit more extensive.

Ideally this feature request would result in a function or tool that could take the old checkpoint and some information about the old + new optimizers, and replace the old checkpoint with the new. This should involve some variable renaming, and some new variables added in. (Hyperparameters are now variables, so we will need to add those in to the checkpoints, and set with reasonable values-- either the defaults, or with user-specified values.)

Priority:
P0 - optimizers
P1 - RNNs
P1 - shared embeddings

[zanieb]

I'm looking into this, seems reasonably straightforward.

[lc0]

I am also happy to help, @dynamicwebpaige please lemme know if this issue is still open

[Gurpreetsingh9465]

This Issue Looks Interesting. Anyone working on this issue? Would love to work on this.😄

[zanieb]

I have been busier than I imagined, so I've just done some research and haven't started working on it. If the people above that are interested want to start outlining what such a tool would look like I'm happy to collaborate on a solution. Alternatively, I won't be offended if someone just does the whole thing. I'll continue to post as I make progress.

[dynamicwebpaige]

This issue is still up for grabs, and we'd appreciate the help! Supporting information for building the conversion mapping can be found in the two referenced issues (#26350, #26349).

[Gurpreetsingh9465]

@dynamicwebpaige am i in a right direction.
we are supposed to build a function model.load_weight(convert('old_checkpoint_path'))
that convert function should change the checkpoint data format into the format supported by tf.keras.optimizer.Adam. as tf.train.AdamOptimizer is deprecated in tf2.0.

cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,verbose=1)

model = create_model(tf.train.AdamOptimizer,0.01) // model trained using tf.train.AdamOptimizer


model.fit(train_images, train_labels,  epochs = 10, 
          validation_data = (test_images,test_labels),
          callbacks = [cp_callback]) // model checkpoint saved

loss, acc = model.evaluate(test_images, test_labels)
print("Trained model, accuracy: {:5.2f}%".format(100*acc))

model = create_model(tf.keras.optimizers.Adam,0.01)

model.load_weights(checkpoint_path) // will cause problem as the format used to save weights by tf.keras.optimizer and tf.train.AdamOptimizer are different

loss, acc = model.evaluate(test_images, test_labels)
print("Untrained model, accuracy: {:5.2f}%".format(100*acc))```

[goldiegadde]

for the P0 item, A checkpoint converter tool for converting optimizers is included with the 2.0.0-beta0 release.

This checkpoint converter tool is mainly for Canned Estimators, including DNN
Linear and DNNLinearCombined estimators. The allowed optimizers to be converted
include Adam, Adagrad, Ftrl, RMSProp, and SGD.

Note that, this converter is not suitable for the case where 'dnn_optimizer'
and 'linear_optimizer' in DNNLinearCombined model are the same.

If your current canned estimators and checkpoints are from TF 1.x, after you
migrate the canned estimator to v2 with tf.keras.optimizers.*, the converted
checkpoint allow you to restore and retrain the model in TF 2.0.

[jvishnuvardhan]

I think this was resolved already with the detailed guide on migrating checkpoints from TF1.x to 2.x.
Here is the link to the guide.

I am closing this issue as this was already resolved. Please feel free to reopen if I am mistaken. Thanks!