Resuming training from saved checkpoint produces different result than uninterrupted training

[lostmsu]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes (Google Collab)
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Collab
• TensorFlow installed from (source or binary): Google Collab
• TensorFlow version (use command below): 1.13.1
• Python version: Python 3

Describe the current behavior
Loading a model with tf.keras.models.load_model, produced with tf.keras.callbacks.ModelCheckpoint , and resuming training produces different results from running the training without save model + restore model in the middle.

Describe the expected behavior
Saving and restoring the model should allow to resume training as if there was no interruption in the first place.

Code to reproduce the issue
Google Collab

Other info / logs
No interruption:

Epoch 49/100

• 0s - loss: 3.5190 - val_loss: 3.3597
  Epoch 50/100
• 0s - loss: 3.4090 - val_loss: 3.2668
  Epoch 51/100
• 0s - loss: 3.2637 - val_loss: 3.1623
  Epoch 52/100
• 0s - loss: 3.0962 - val_loss: 2.9975

With interruption:

Epoch 49/50

• 0s - loss: 3.5190 - val_loss: 3.3597
  Epoch 50/50

Epoch 00050: saving model to weights.50.ckpt

• 0s - loss: 3.4090 - val_loss: 3.2668
  ... load_model('weights.50.ckpt') ...
  Epoch 51/100
• 0s - loss: 3.2637 - val_loss: 3.3816
  Epoch 52/100
• 0s - loss: 3.3175 - val_loss: 3.1457

The model does not have any random elements, so it looks like optimizer state is lost.

[allenlavoie]

Model.fit is taking random slices of the data and batching them together. If you control for that, e.g. make all of the examples identical, can you still reproduce?

[lostmsu]

@allenlavoie the model will converge instantly if there's just 1 training sample.
I added shuffle=False to .fit(...) calls. The issue persisted. (notebook updated).

Also, this should not have changed anything at all, since Has no effect when steps_per_epoch is not None., which was 1 in my original sample anyway. Indeed, the losses after this change are the same.

[allenlavoie]

Hrm. It looks like it's passing include_optimizer to Model.save. @tanzhenyu, any ideas?

[tanzhenyu]

@lostmsu The previous keras optimizer is problematic. We have provided a new set of optimizers under exactly the same name and fully backward compatible for users. However as unfortunate as this is, the new ones did not make to tf 1.13.1. If you do !pip install tf-nightly-gpu-2.0-preview, then you should be able to see identical behavior.

That said, you need to change one line from tf.set_random_seed(seed) to tf.random.set_seed(seed)

[lostmsu]

@tanzhenyu confirming, seems to be fixed in 2.0-preview

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No