save_and_restore_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g_nWetWWd_ns"
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "2pHVBk_seED1"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "N_fMsQ-N8I7j"
   },
   "outputs": [],
   "source": [
    "#@title MIT License\n",
    "#\n",
    "# Copyright (c) 2017 François Chollet\n",
    "#\n",
    "# Permission is hereby granted, free of charge, to any person obtaining a\n",
    "# copy of this software and associated documentation files (the \"Software\"),\n",
    "# to deal in the Software without restriction, including without limitation\n",
    "# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
    "# and/or sell copies of the Software, and to permit persons to whom the\n",
    "# Software is furnished to do so, subject to the following conditions:\n",
    "#\n",
    "# The above copyright notice and this permission notice shall be included in\n",
    "# all copies or substantial portions of the Software.\n",
    "#\n",
    "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
    "# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
    "# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
    "# DEALINGS IN THE SOFTWARE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pZJ3uY9O17VN"
   },
   "source": [
    "# Save and restore models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M4Ata7_wMul1"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/beta/tutorials/keras/save_and_restore_models\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/keras/save_and_restore_models.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/keras/save_and_restore_models.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/r2/tutorials/keras/save_and_restore_models.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mBdde4YJeJKF"
   },
   "source": [
    "Model progress can be saved during—and after—training. This means a model can resume where it left off and avoid long training times. Saving also means you can share your model and others can recreate your work. When publishing research models and techniques, most machine learning practitioners share:\n",
    "\n",
    "* code to create the model, and\n",
    "* the trained weights, or parameters, for the model\n",
    "\n",
    "Sharing this data helps others understand how the model works and try it themselves with new data.\n",
    "\n",
    "Caution: Be careful with untrusted code—TensorFlow models are code. See [Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md) for details.\n",
    "\n",
    "### Options\n",
    "\n",
    "There are different ways to save TensorFlow models—depending on the API you're using. This guide uses [tf.keras](https://www.tensorflow.org/guide/keras), a high-level API to build and train models in TensorFlow. For other approaches, see the TensorFlow  [Save and Restore](https://www.tensorflow.org/guide/saved_model) guide or [Saving in eager](https://www.tensorflow.org/guide/eager#object-based_saving).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xCUREq7WXgvg"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SbGsznErXWt6"
   },
   "source": [
    "### Get an example dataset\n",
    "\n",
    "We'll use the [MNIST dataset](http://yann.lecun.com/exdb/mnist/) to train our model to demonstrate saving weights. To speed up these demonstration runs, only use the first 1000 examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tell TF not to consume all GPU memory so you can run more than one notebook at once\n",
    "import os\n",
    "os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[37mcgpu03\u001b[m  Sat Jul 13 16:28:25 2019\n",
      "\u001b[36m[0]\u001b[m \u001b[34mTesla V100-SXM2-16GB\u001b[m |\u001b[31m 36'C\u001b[m, \u001b[32m  0 %\u001b[m | \u001b[36m\u001b[1m\u001b[33m    0\u001b[m / \u001b[33m16130\u001b[m MB |\n"
     ]
    }
   ],
   "source": [
    "!gpustat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7Nm7Tyb-gRt-"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0-beta1'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9rGfFwE9XVwz"
   },
   "outputs": [],
   "source": [
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "train_labels = train_labels[:1000]\n",
    "test_labels = test_labels[:1000]\n",
    "\n",
    "train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0\n",
    "test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "anG3iVoXyZGI"
   },
   "source": [
    "### Define a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wynsOBfby0Pa"
   },
   "source": [
    "Let's build a simple model we'll use to demonstrate saving and loading weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0HZbJIjxyX1S"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Returns a short sequential model\n",
    "def create_model():\n",
    "    model = tf.keras.models.Sequential([\n",
    "        keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n",
    "        keras.layers.Dropout(0.2),\n",
    "        keras.layers.Dense(10, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "# Create a basic model instance\n",
    "model = create_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "soDE0W_KH8rG"
   },
   "source": [
    "## Save checkpoints during training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mRyd5qQQIXZm"
   },
   "source": [
    "The primary use case is to automatically save checkpoints *during* and at *the end* of training. This way you can use a trained model without having to retrain it, or pick-up training where you left of—in case the training process was interrupted.\n",
    "\n",
    "`tf.keras.callbacks.ModelCheckpoint` is a callback that performs this task. The callback takes a couple of arguments to configure checkpointing.\n",
    "\n",
    "### Checkpoint callback usage\n",
    "\n",
    "Train the model and pass it the `ModelCheckpoint` callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IFPuhwntH8VH"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0713 16:29:02.212527 46912496740800 deprecation.py:323] From /usr/common/software/tensorflow/gpu-tensorflow/2.0.0-beta-py36/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples, validate on 1000 samples\n",
      "Epoch 1/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 1.2869 - accuracy: 0.6214 \n",
      "Epoch 00001: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 1s 641us/sample - loss: 1.1847 - accuracy: 0.6510 - val_loss: 0.7144 - val_accuracy: 0.7840\n",
      "Epoch 2/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.4252 - accuracy: 0.8690\n",
      "Epoch 00002: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 123us/sample - loss: 0.4301 - accuracy: 0.8700 - val_loss: 0.5157 - val_accuracy: 0.8430\n",
      "Epoch 3/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.2988 - accuracy: 0.9255\n",
      "Epoch 00003: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 117us/sample - loss: 0.2923 - accuracy: 0.9270 - val_loss: 0.4626 - val_accuracy: 0.8530\n",
      "Epoch 4/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.2020 - accuracy: 0.9615\n",
      "Epoch 00004: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 117us/sample - loss: 0.2067 - accuracy: 0.9590 - val_loss: 0.4642 - val_accuracy: 0.8520\n",
      "Epoch 5/10\n",
      " 800/1000 [=======================>......] - ETA: 0s - loss: 0.1493 - accuracy: 0.9700\n",
      "Epoch 00005: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 123us/sample - loss: 0.1578 - accuracy: 0.9680 - val_loss: 0.4094 - val_accuracy: 0.8630\n",
      "Epoch 6/10\n",
      " 800/1000 [=======================>......] - ETA: 0s - loss: 0.1092 - accuracy: 0.9800\n",
      "Epoch 00006: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 121us/sample - loss: 0.1173 - accuracy: 0.9780 - val_loss: 0.4104 - val_accuracy: 0.8650\n",
      "Epoch 7/10\n",
      " 800/1000 [=======================>......] - ETA: 0s - loss: 0.0867 - accuracy: 0.9887\n",
      "Epoch 00007: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 118us/sample - loss: 0.0874 - accuracy: 0.9870 - val_loss: 0.4074 - val_accuracy: 0.8570\n",
      "Epoch 8/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.0591 - accuracy: 0.9964\n",
      "Epoch 00008: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 117us/sample - loss: 0.0602 - accuracy: 0.9950 - val_loss: 0.4031 - val_accuracy: 0.8560\n",
      "Epoch 9/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.0586 - accuracy: 0.9952\n",
      "Epoch 00009: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 117us/sample - loss: 0.0562 - accuracy: 0.9960 - val_loss: 0.4018 - val_accuracy: 0.8630\n",
      "Epoch 10/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.0404 - accuracy: 0.9988\n",
      "Epoch 00010: saving model to training_1/cp.ckpt\n",
      "1000/1000 [==============================] - 0s 117us/sample - loss: 0.0428 - accuracy: 0.9990 - val_loss: 0.4028 - val_accuracy: 0.8680\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2aab1cf1b978>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "checkpoint_path = \"training_1/cp.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "# Create checkpoint callback\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 verbose=1)\n",
    "\n",
    "model = create_model()\n",
    "\n",
    "model.fit(train_images, train_labels,  epochs = 10,\n",
    "          validation_data = (test_images,test_labels),\n",
    "          callbacks = [cp_callback])  # pass callback to training\n",
    "\n",
    "# This may generate warnings related to saving the state of the optimizer.\n",
    "# These warnings (and similar warnings throughout this notebook)\n",
    "# are in place to discourage outdated usage, and can be ignored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rlM-sgyJO084"
   },
   "source": [
    "This creates a single collection of TensorFlow checkpoint files that are updated at the end of each epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gXG5FVKFOVQ3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint\t\t     cp.ckpt.data-00001-of-00002\n",
      "cp.ckpt.data-00000-of-00002  cp.ckpt.index\n"
     ]
    }
   ],
   "source": [
    "!ls {checkpoint_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wlRN_f56Pqa9"
   },
   "source": [
    "Create a new, untrained model. When restoring a model from only weights, you must have a model with the same architecture as the original model. Since it's the same model architecture, we can share weights despite that it's a different *instance* of the model.\n",
    "\n",
    "Now rebuild a fresh, untrained model, and evaluate it on the test set. An untrained model will perform at chance levels (~10% accuracy):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fp5gbuiaPqCT"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 92us/sample - loss: 2.3596 - accuracy: 0.1370\n",
      "Untrained model, accuracy: 13.70%\n"
     ]
    }
   ],
   "source": [
    "model = create_model()\n",
    "\n",
    "loss, acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Untrained model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1DTKpZssRSo3"
   },
   "source": [
    "Then load the weights from the checkpoint, and re-evaluate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2IZxbwiRRSD2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 35us/sample - loss: 0.4028 - accuracy: 0.8680\n",
      "Restored model, accuracy: 86.80%\n"
     ]
    }
   ],
   "source": [
    "model.load_weights(checkpoint_path)\n",
    "loss,acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bpAbKkAyVPV8"
   },
   "source": [
    "### Checkpoint callback options\n",
    "\n",
    "The callback provides several options to give the resulting checkpoints unique names, and adjust the checkpointing frequency.\n",
    "\n",
    "Train a new model, and save uniquely named checkpoints once every 5-epochs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mQF_dlgIVOvq"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0713 16:29:51.184920 46912496740800 callbacks.py:859] `period` argument is deprecated. Please use `save_freq` to specify the frequency in number of samples seen.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 00005: saving model to training_2/cp-0005.ckpt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0713 16:29:52.894087 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.iter\n",
      "W0713 16:29:52.894643 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_1\n",
      "W0713 16:29:52.897105 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_2\n",
      "W0713 16:29:52.897518 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.decay\n",
      "W0713 16:29:52.897911 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
      "W0713 16:29:52.898756 46912496740800 util.py:252] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 00010: saving model to training_2/cp-0010.ckpt\n",
      "\n",
      "Epoch 00015: saving model to training_2/cp-0015.ckpt\n",
      "\n",
      "Epoch 00020: saving model to training_2/cp-0020.ckpt\n",
      "\n",
      "Epoch 00025: saving model to training_2/cp-0025.ckpt\n",
      "\n",
      "Epoch 00030: saving model to training_2/cp-0030.ckpt\n",
      "\n",
      "Epoch 00035: saving model to training_2/cp-0035.ckpt\n",
      "\n",
      "Epoch 00040: saving model to training_2/cp-0040.ckpt\n",
      "\n",
      "Epoch 00045: saving model to training_2/cp-0045.ckpt\n",
      "\n",
      "Epoch 00050: saving model to training_2/cp-0050.ckpt\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2aab1ee6c518>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# include the epoch in the file name. (uses `str.format`)\n",
    "checkpoint_path = \"training_2/cp-{epoch:04d}.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n",
    "    checkpoint_path, verbose=1, save_weights_only=True,\n",
    "    # Save weights, every 5-epochs.\n",
    "    period=5)\n",
    "\n",
    "model = create_model()\n",
    "model.save_weights(checkpoint_path.format(epoch=0))\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs = 50, callbacks = [cp_callback],\n",
    "          validation_data = (test_images,test_labels),\n",
    "          verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1zFrKTjjavWI"
   },
   "source": [
    "Now, look at the resulting checkpoints and choose the latest one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p64q3-V4sXt0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint\t\t\t  cp-0025.ckpt.data-00001-of-00002\n",
      "cp-0000.ckpt.data-00000-of-00002  cp-0025.ckpt.index\n",
      "cp-0000.ckpt.data-00001-of-00002  cp-0030.ckpt.data-00000-of-00002\n",
      "cp-0000.ckpt.index\t\t  cp-0030.ckpt.data-00001-of-00002\n",
      "cp-0005.ckpt.data-00000-of-00002  cp-0030.ckpt.index\n",
      "cp-0005.ckpt.data-00001-of-00002  cp-0035.ckpt.data-00000-of-00002\n",
      "cp-0005.ckpt.index\t\t  cp-0035.ckpt.data-00001-of-00002\n",
      "cp-0010.ckpt.data-00000-of-00002  cp-0035.ckpt.index\n",
      "cp-0010.ckpt.data-00001-of-00002  cp-0040.ckpt.data-00000-of-00002\n",
      "cp-0010.ckpt.index\t\t  cp-0040.ckpt.data-00001-of-00002\n",
      "cp-0015.ckpt.data-00000-of-00002  cp-0040.ckpt.index\n",
      "cp-0015.ckpt.data-00001-of-00002  cp-0045.ckpt.data-00000-of-00002\n",
      "cp-0015.ckpt.index\t\t  cp-0045.ckpt.data-00001-of-00002\n",
      "cp-0020.ckpt.data-00000-of-00002  cp-0045.ckpt.index\n",
      "cp-0020.ckpt.data-00001-of-00002  cp-0050.ckpt.data-00000-of-00002\n",
      "cp-0020.ckpt.index\t\t  cp-0050.ckpt.data-00001-of-00002\n",
      "cp-0025.ckpt.data-00000-of-00002  cp-0050.ckpt.index\n"
     ]
    }
   ],
   "source": [
    "! ls {checkpoint_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1AN_fnuyR41H"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_2/cp-0050.ckpt'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "latest = tf.train.latest_checkpoint(checkpoint_dir)\n",
    "latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zk2ciGbKg561"
   },
   "source": [
    "Note: the default tensorflow format only saves the 5 most recent checkpoints.\n",
    "\n",
    "To test, reset the model and load the latest checkpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3M04jyK-H3QK"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 90us/sample - loss: 0.5014 - accuracy: 0.8730\n",
      "Restored model, accuracy: 87.30%\n"
     ]
    }
   ],
   "source": [
    "model = create_model()\n",
    "model.load_weights(latest)\n",
    "loss, acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c2OxsJOTHxia"
   },
   "source": [
    "## What are these files?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JtdYhvWnH2ib"
   },
   "source": [
    "The above code stores the weights to a collection of [checkpoint](https://www.tensorflow.org/guide/saved_model#save_and_restore_variables)-formatted files that contain only the trained weights in a binary format. Checkpoints contain:\n",
    "* One or more shards that contain your model's weights.\n",
    "* An index file that indicates which weights are stored in a which shard.\n",
    "\n",
    "If you are only training a model on a single machine, you'll have one shard with the suffix: `.data-00000-of-00001`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S_FA-ZvxuXQV"
   },
   "source": [
    "## Manually save weights\n",
    "\n",
    "Above you saw how to load the weights into a model.\n",
    "\n",
    "Manually saving the weights is just as simple, use the `Model.save_weights` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R7W5plyZ-u9X"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 92us/sample - loss: 0.5014 - accuracy: 0.8730\n",
      "Restored model, accuracy: 87.30%\n"
     ]
    }
   ],
   "source": [
    "# Save the weights\n",
    "model.save_weights('./checkpoints/my_checkpoint')\n",
    "\n",
    "# Restore the weights\n",
    "model = create_model()\n",
    "model.load_weights('./checkpoints/my_checkpoint')\n",
    "\n",
    "loss,acc = model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kOGlxPRBEvV1"
   },
   "source": [
    "## Save the entire model\n",
    "\n",
    "The model and optimizer can be saved to a file that contains both their state (weights and variables), and the model configuration. This allows you to export a model so it can be used without access to the original python code. Since the optimizer-state is recovered you can even resume training from exactly where you left off.\n",
    "\n",
    "Saving a fully-functional model is very useful—you can load them in TensorFlow.js ([HDF5](https://js.tensorflow.org/tutorials/import-keras.html), [Saved Model](https://js.tensorflow.org/tutorials/import-saved-model.html)) and then train and run them in web browsers, or convert them to run on mobile devices using TensorFlow Lite ([HDF5](https://www.tensorflow.org/lite/convert/python_api#exporting_a_tfkeras_file_), [Saved Model](https://www.tensorflow.org/lite/convert/python_api#exporting_a_savedmodel_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SkGwf-50zLNn"
   },
   "source": [
    "### As an HDF5 file\n",
    "\n",
    "Keras provides a basic save format using the [HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) standard. For our purposes, the saved model can be treated as a single binary blob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m2dkmJVCGUia"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 228us/sample - loss: 1.1308 - accuracy: 0.6570\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 0.4309 - accuracy: 0.8740\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 0.2839 - accuracy: 0.9320\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 64us/sample - loss: 0.2064 - accuracy: 0.9460\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 0.1507 - accuracy: 0.9690\n"
     ]
    }
   ],
   "source": [
    "model = create_model()\n",
    "\n",
    "model.fit(train_images, train_labels, epochs=5)\n",
    "\n",
    "# Save entire model to a HDF5 file\n",
    "model.save('my_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GWmttMOqS68S"
   },
   "source": [
    "Now recreate the model from that file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5NDMO_7kS6Do"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0713 16:34:17.752168 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.iter\n",
      "W0713 16:34:17.752822 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_1\n",
      "W0713 16:34:17.753264 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_2\n",
      "W0713 16:34:17.754821 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.decay\n",
      "W0713 16:34:17.756290 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
      "W0713 16:34:17.757513 46912496740800 util.py:252] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_6\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_12 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Recreate the exact same model, including weights and optimizer.\n",
    "new_model = keras.models.load_model('my_model.h5')\n",
    "new_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JXQpbTicTBwt"
   },
   "source": [
    "Check its accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jwEaj9DnTCVA"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 95us/sample - loss: 0.4257 - accuracy: 0.8540\n",
      "Restored model, accuracy: 85.40%\n"
     ]
    }
   ],
   "source": [
    "loss, acc = new_model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dGXqd4wWJl8O"
   },
   "source": [
    "This technique saves everything:\n",
    "\n",
    "* The weight values\n",
    "* The model's configuration(architecture)\n",
    "* The optimizer configuration\n",
    "\n",
    "Keras saves models by inspecting the architecture. Currently, it is not able to save TensorFlow optimizers (from `tf.train`). When using those you will need to re-compile the model after loading, and you will lose the state of the optimizer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kPyhgcoVzqUB"
   },
   "source": [
    "### As a `saved_model`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LtcN4VIb7JkK"
   },
   "source": [
    "Caution: This method of saving a `tf.keras` model is experimental and may change in future versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DSWiSB0Q8c46"
   },
   "source": [
    "Build a fresh model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sI1YvCDFzpl3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 237us/sample - loss: 1.1295 - accuracy: 0.6870\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 72us/sample - loss: 0.4160 - accuracy: 0.8890\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 67us/sample - loss: 0.2783 - accuracy: 0.9330\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 0.2048 - accuracy: 0.9510\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 0.1506 - accuracy: 0.9690\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2aab1fe3ea20>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = create_model()\n",
    "\n",
    "model.fit(train_images, train_labels, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iUvT_3qE8hV5"
   },
   "source": [
    "Create a `saved_model`, and place it in a time-stamped directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sq8fPglI1RWA"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0713 16:34:53.631388 46912496740800 deprecation.py:323] From /usr/common/software/tensorflow/gpu-tensorflow/2.0.0-beta-py36/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:253: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "W0713 16:34:53.633486 46912496740800 export_utils.py:182] Export includes no default signature!\n",
      "W0713 16:34:53.839863 46912496740800 export_utils.py:182] Export includes no default signature!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'./saved_models/1563060893'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "saved_model_path = \"./saved_models/{}\".format(int(time.time()))\n",
    "\n",
    "tf.keras.experimental.export_saved_model(model, saved_model_path)\n",
    "saved_model_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MjpmyPfh8-1n"
   },
   "source": [
    "List your saved models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZtOvxA7V0iTv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1563060893\n"
     ]
    }
   ],
   "source": [
    "!ls saved_models/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B7qfpvpY9HCe"
   },
   "source": [
    "Reload a fresh keras model from the saved model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0YofwHdN0pxa"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0713 16:35:09.242412 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_1\n",
      "W0713 16:35:09.243044 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_2\n",
      "W0713 16:35:09.243519 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.decay\n",
      "W0713 16:35:09.243970 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
      "W0713 16:35:09.244395 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.kernel\n",
      "W0713 16:35:09.244815 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.bias\n",
      "W0713 16:35:09.245213 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.kernel\n",
      "W0713 16:35:09.245635 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.bias\n",
      "W0713 16:35:09.246025 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.kernel\n",
      "W0713 16:35:09.246461 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.bias\n",
      "W0713 16:35:09.250754 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.kernel\n",
      "W0713 16:35:09.251188 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.bias\n",
      "W0713 16:35:09.251616 46912496740800 util.py:252] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.\n",
      "W0713 16:35:09.254225 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer\n",
      "W0713 16:35:09.254666 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.iter\n",
      "W0713 16:35:09.255103 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_1\n",
      "W0713 16:35:09.255517 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_2\n",
      "W0713 16:35:09.255945 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.decay\n",
      "W0713 16:35:09.256351 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
      "W0713 16:35:09.262776 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.kernel\n",
      "W0713 16:35:09.263218 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.bias\n",
      "W0713 16:35:09.263681 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.kernel\n",
      "W0713 16:35:09.264089 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.bias\n",
      "W0713 16:35:09.264487 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.kernel\n",
      "W0713 16:35:09.264889 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.bias\n",
      "W0713 16:35:09.265260 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.kernel\n",
      "W0713 16:35:09.265689 46912496740800 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.bias\n",
      "W0713 16:35:09.266100 46912496740800 util.py:252] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_7\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_14 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "new_model = tf.keras.experimental.load_from_saved_model(saved_model_path)\n",
    "new_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uWwgNaz19TH2"
   },
   "source": [
    "Run the restored model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yh5Mu0yOgE5J"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 10)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(test_images).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pc9e6G6w1AWG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 90us/sample - loss: 0.4166 - accuracy: 0.8640\n",
      "Restored model, accuracy: 86.40%\n"
     ]
    }
   ],
   "source": [
    "# The model has to be compiled before evaluating.\n",
    "# This step is not required if the saved model is only being deployed.\n",
    "\n",
    "new_model.compile(optimizer=model.optimizer,  # keep the optimizer that was loaded\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Evaluate the restored model.\n",
    "loss, acc = new_model.evaluate(test_images, test_labels)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eUYTzSz5VxL2"
   },
   "source": [
    "## What's Next\n",
    "\n",
    "That was a quick guide to saving and loading in with `tf.keras`.\n",
    "\n",
    "* The [tf.keras guide](https://www.tensorflow.org/guide/keras) shows more about saving and loading models with `tf.keras`.\n",
    "\n",
    "* See [Saving in eager](https://www.tensorflow.org/guide/eager#object_based_saving) for saving during eager execution.\n",
    "\n",
    "* The [Save and Restore](https://www.tensorflow.org/guide/saved_model) guide has low-level details about TensorFlow saving."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "save_and_restore_models.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "tensorflow-gpu/2.0.0-beta-py36",
   "language": "python",
   "name": "tensorflow_gpu_2.0.0-beta-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}