Accuracy is lost after save/load

[ThatDockerUser]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): See below.
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10, 1909
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: -
• TensorFlow installed from (source or binary): pip
• TensorFlow version (use command below): v2.3.0-rc2-23-gb36436b087 2.3.0
• Python version: 3.7.7
• Bazel version (if compiling from source): -
• GCC/Compiler version (if compiling from source): -
• CUDA/cuDNN version: -
• GPU model and memory: Not relevant.

Describe the current behavior
When saving a pre-trained model and loading it again, the accuracy drops to its default value. Minimal example:

import tensorflow as tf

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=2, activation='softmax', name='output'))
model.compile(optimizer=tf.keras.optimizers.Adam(lr=10),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
dummy_data_x = [[0, 0],
                [1, 0],
                [0, 1],
                [1, 1]]
dummy_data_y = [0, 1, 0, 1]
print(model.evaluate(x=dummy_data_x, y=dummy_data_y))
model.fit(x=dummy_data_x, y=dummy_data_y, epochs=10)
print(model.evaluate(x=dummy_data_x, y=dummy_data_y))
model.save('test_model')
model = tf.keras.models.load_model('test_model')
print(model.evaluate(x=dummy_data_x, y=dummy_data_y))

The model is extremely simple (read: bad/useless) for comparison's sake. Before training it, the evaluation results in:

1/1 [==============================] - 0s 0s/step - loss: 0.9013 - accuracy: 0.5000
[0.9013183116912842, 0.5]

The loss is obviously random at first, but crucially the accuracy is 50% because it guesses 0 every time. After training, it evaluates to:

1/1 [==============================] - 0s 0s/step - loss: 0.0000e+00 - accuracy: 1.0000
[0.0, 1.0]

The loss dropped to zero and the model can perfectly interpret the data. Now I save the model to the disk and immediately load it from the same location. When I now evaluate it, I get:

1/1 [==============================] - 0s 1000us/step - loss: 0.0000e+00 - accuracy: 0.5000
[0.0, 0.5]

Even though the model has the same structure, weights, and loss, and all four example inputs are evaluated correctly, TensorFlow says the accuracy is 50%, which is not true.

Describe the expected behavior
The accuracy should remain the same when loading a previously trained and saved model.

Standalone code to reproduce the issue
See above.

Other info / logs
See above.

[ravikyram]

I have tried in colab with TF version 2.3, nightly version(2.4.0-dev20200818) and was able to reproduce the issue.Please, find the gist here. Thanks!

[jvishnuvardhan]

@ThatDockerUser This is a known issue. Team is working on correcting it.

Please check the response from @k-w-w

This is a bug with using the sparse categorical accuracy. For now, please compile the model with metrics='sparse_categorical_accuracy' instead of just 'accuracy'.

With the above change (workaround), the results are same before saving and after loading the model back. Please check the gist here.

We will follow the progress with that previous issue. Thanks!

[ThatDockerUser]

@jvishnuvardhan Thanks for the reply. The SC-accuracy does appear to work. I'm curious, though: How is it different from "regular" accuracy?

[jvishnuvardhan]

@ThatDockerUser Based on loss function you defined in the model.compile, under the hood TF will select appropriate accuracy method. In your case, it should select 'sparse_categorical_accuracy` but currently there is a known bug. It will be corrected soon. Until then please specify explicitly as mentioned above. Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[goldiegadde]

This issue has been fixed in commit 9d8947. Please try the latest tf-nightly if you need the fix immediately, otherwise the next official TF release will have the fix. Marking this as closed.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[akbaramed]

As on 20th Oct 2020.
if anyone is still looking around, using python 3.610.
Model.save and load_model not working for keras 2.3.1 and tensor flow 2.2.0
using model.to_json and models.model_from_json also doesn't work.
The loaded model has definitely something missing and gives way lesser accuracy as the saved model in a different session/spyder consoles.
upgrade the packages to keras 2.4.3 and tensorflow 2.3.0. together.

[jvishnuvardhan]

@akbaramed As mentioned above, the bug was corrected and we can no longer face the issue when you use tf-nightly. If you have Tensorflow 2.3, then please follow the workaround `metrics='sparse_categorical_accuracy' instead of just 'accuracy'.

In the near future, stable TF2.4 will be released. Thanks!

If this is still an issue with tf-nightly, please provide a standalone code to reproduce the issue. Thanks!