SaveAndLoadModels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save and load models\n",
    "Tensorflow 2.0에서는 Keras 사용을 권장하고 사용하게 된다.  \n",
    "이번 Post에서는 실제로 Training된 Model을 Save하고 Load하는 방법에 대해서 다룬다.  \n",
    "기본적으로 <a href=\"https://wjddyd66.github.io/categories/#keras\">Keras Category</a>에서 Model을 저장하고 불러오는 방법과 <a href=\"https://wjddyd66.github.io/keras/Keras(5)/#%EC%BC%80%EB%9D%BC%EC%8A%A4-%EC%BD%9C%EB%B0%B1%EA%B3%BC-%ED%85%90%EC%84%9C%EB%B3%B4%EB%93%9C%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%9C-%EB%94%A5%EB%9F%AC%EB%8B%9D-%EB%AA%A8%EB%8D%B8-%EA%B2%80%EC%82%AC%EC%99%80-%EB%AA%A8%EB%8B%88%ED%84%B0%EB%A7%81\">Keras Callback</a>에서 Keras의 Callback에 대한 사전지식이 있으면 수월하게 넘어갈 수 있는 Post이다.  \n",
    "\n",
    "사전 사항으로서 pyyaml, h5py 2개의 Python Package를 설치하여야 한다.  \n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 필요한 Library Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import os\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "print(tf.version.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get an example dataset\n",
    "Keras의 <code>tf.keras.datasets.mnist.load_data()</code>를 활용하여 Mnist Dataset을 다운받는다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mnist Dataset Download\n",
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Mnist Dataset Indexing\n",
    "train_labels = train_labels[:1000]\n",
    "test_labels = test_labels[:1000]\n",
    "\n",
    "# Dataset Normalization\n",
    "train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0\n",
    "test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a model\n",
    "실제로 Save and Load할 Base Model을 선언한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define a simple sequential model\n",
    "def create_model():\n",
    "    model = tf.keras.models.Sequential([\n",
    "        keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n",
    "        keras.layers.Dropout(0.2),\n",
    "        keras.layers.Dense(10, activation='softmax')\n",
    "    ])\n",
    "    \n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model\n",
    "\n",
    "# Create a basic model instance\n",
    "model = create_model()\n",
    "\n",
    "# Display the model's architecture\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save checkpoints during training\n",
    "Keras의 Callback을 사용하여 Model이 Training되는 동안 Checkpoints를 저장한다.  \n",
    "Keras의 Callback에 대한 사전 지신은 링크를 참조하자. <a href=\"https://wjddyd66.github.io/keras/Keras(5)/#%EC%BC%80%EB%9D%BC%EC%8A%A4-%EC%BD%9C%EB%B0%B1%EA%B3%BC-%ED%85%90%EC%84%9C%EB%B3%B4%EB%93%9C%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%9C-%EB%94%A5%EB%9F%AC%EB%8B%9D-%EB%AA%A8%EB%8D%B8-%EA%B2%80%EC%82%AC%EC%99%80-%EB%AA%A8%EB%8B%88%ED%84%B0%EB%A7%81\">Keras Callback</a>  \n",
    "Keras의 Callback 중 <code>tf.keras.callbacks.ModelCheckpoint()</code>를 사용한다.  \n",
    "\n",
    "**tf.keras.callbacks.ModelCheckpoint() Argument**  \n",
    "- filepath: Model file을 저장할 경로\n",
    "- monitor: Monitor할 수량\n",
    "- verbose: 0 or 1 Training되는 동안 상황을 지켜볼 것인지 아닌지\n",
    "- save_best_only: 가장 성능이 좋은 Model File만 저장한다.\n",
    "- save_weights_only: True이면 모델의 가중치만 저장한다.\n",
    "\n",
    "위의 Argument를 제외하고 많은 Option을 제공한다. 다양한 Option과 자세한 사용법은 링크를 참조하자.  \n",
    "참조: <a href=\"https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint?version=stable\">tf.keras.callbacks.ModelCheckpoint() 사용법</a>  \n",
    "\n",
    "아래 Code를 살펴보면 다음과 같다.  \n",
    "<code>tf.keras.callbacks.ModelCheckpoint()</code>: Keras Callback Object 선언\n",
    "- filepath: Model File이 저장될 경로\n",
    "- verbose = 1: Training 중 매 Epoch마다 확인\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples, validate on 1000 samples\n",
      "Epoch 1/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000    \n",
      "Epoch 00001: saving model to training_1/cp.ckpt\n",
      "WARNING:tensorflow:From /root/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 1s 690us/sample - loss: 0.0011 - accuracy: 1.0000 - val_loss: 0.4999 - val_accuracy: 0.8800\n",
      "Epoch 2/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000    \n",
      "Epoch 00002: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 444us/sample - loss: 0.0012 - accuracy: 1.0000 - val_loss: 0.4922 - val_accuracy: 0.8800\n",
      "Epoch 3/10\n",
      " 768/1000 [======================>.......] - ETA: 0s - loss: 9.8459e-04 - accuracy: 1.0000\n",
      "Epoch 00003: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 375us/sample - loss: 0.0010 - accuracy: 1.0000 - val_loss: 0.5035 - val_accuracy: 0.8810\n",
      "Epoch 4/10\n",
      " 800/1000 [=======================>......] - ETA: 0s - loss: 9.9316e-04 - accuracy: 1.0000\n",
      "Epoch 00004: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 379us/sample - loss: 0.0010 - accuracy: 1.0000 - val_loss: 0.5054 - val_accuracy: 0.8800\n",
      "Epoch 5/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 9.8397e-04 - accuracy: 1.0000\n",
      "Epoch 00005: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 447us/sample - loss: 9.6179e-04 - accuracy: 1.0000 - val_loss: 0.5033 - val_accuracy: 0.8790\n",
      "Epoch 6/10\n",
      " 928/1000 [==========================>...] - ETA: 0s - loss: 8.8455e-04 - accuracy: 1.0000\n",
      "Epoch 00006: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 371us/sample - loss: 8.8179e-04 - accuracy: 1.0000 - val_loss: 0.5020 - val_accuracy: 0.8780\n",
      "Epoch 7/10\n",
      " 864/1000 [========================>.....] - ETA: 0s - loss: 8.8211e-04 - accuracy: 1.0000\n",
      "Epoch 00007: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 378us/sample - loss: 8.7298e-04 - accuracy: 1.0000 - val_loss: 0.5159 - val_accuracy: 0.8780\n",
      "Epoch 8/10\n",
      " 896/1000 [=========================>....] - ETA: 0s - loss: 8.2322e-04 - accuracy: 1.0000\n",
      "Epoch 00008: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 471us/sample - loss: 8.5062e-04 - accuracy: 1.0000 - val_loss: 0.5095 - val_accuracy: 0.8800\n",
      "Epoch 9/10\n",
      " 768/1000 [======================>.......] - ETA: 0s - loss: 9.0811e-04 - accuracy: 1.0000\n",
      "Epoch 00009: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 418us/sample - loss: 8.5639e-04 - accuracy: 1.0000 - val_loss: 0.5047 - val_accuracy: 0.8820\n",
      "Epoch 10/10\n",
      " 832/1000 [=======================>......] - ETA: 0s - loss: 8.5775e-04 - accuracy: 1.0000\n",
      "Epoch 00010: saving model to training_1/cp.ckpt\n",
      "INFO:tensorflow:Assets written to: training_1/cp.ckpt/assets\n",
      "1000/1000 [==============================] - 0s 390us/sample - loss: 8.3009e-04 - accuracy: 1.0000 - val_loss: 0.5114 - val_accuracy: 0.8790\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f2f38fdb910>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Keras CallBack Modelcheckpoint 의 Option 설정\n",
    "checkpoint_path = \"training_1/cp.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "# Keras CallBack 선언\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,verbose=1)\n",
    "\n",
    "# Train the model with the new callback\n",
    "model.fit(train_images, \n",
    "          train_labels,  \n",
    "          epochs=10,\n",
    "          validation_data=(test_images,test_labels),\n",
    "          callbacks=[cp_callback])  # Pass callback to training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위의 Directory의 결과를 확인하면 다음과 같다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint  cp.ckpt  cp.ckpt.data-00000-of-00001  cp.ckpt.index\r\n"
     ]
    }
   ],
   "source": [
    "!ls {checkpoint_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Directory에 저장되는 File의 의미는 다음과 같다.  \n",
    "- data file: it is TensorBundle collection, save the values of all variables.\n",
    "- index file: it is a string-string immutable table(tensorflow::table::Table). Each key is a name of a tensor and its value is a serialized BundleEntryProto. Each BundleEntryProto describes the metadata of a tensor: which of the \"data\" files contains the content of a tensor, the offset into that file, checksum, some auxiliary data, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Model\n",
    "아래 Code는 Weight가 Training되지 않은 Model과 위에서 Training된 Model을 Load하여 Accuracy를 비교하는 Code이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1 - 0s - loss: 2.2989 - accuracy: 0.0780\n",
      "Untrained model, accuracy:  7.80%\n",
      "1000/1 - 0s - loss: 0.4418 - accuracy: 0.8670\n",
      "Untrained model, accuracy: 86.70%\n"
     ]
    }
   ],
   "source": [
    "# Training되지 않은 Model Accuacy 측정\n",
    "model = create_model()\n",
    "loss, acc = model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print(\"Untrained model, accuracy: {:5.2f}%\".format(100*acc))\n",
    "\n",
    "# Training된 Model Accuacy 측정\n",
    "# checkpoint_path = \"training_1/cp.ckpt\"\n",
    "model.load_weights(checkpoint_path)\n",
    "loss, acc = model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print(\"Untrained model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checkpoint callback options\n",
    "아래 Code는 Checkpoint callback options를 추가적으로사용한다.  \n",
    "- <code>checkpoint_path = \"training_2/cp-{epoch:04d}.ckpt\"</code>: Model File의 저장을 str.format() 형태로서 정의할 수 있다. \n",
    "- <code>period=5</code>: Checkpoint callback은 5번의 Epoch마다 수행된다.\n",
    "- <code>tf.train.latest_checkpoint()</code>: 마지막으로 저장된 Model File을 확인할 수 있다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of samples seen.\n",
      "\n",
      "Epoch 00005: saving model to training_2/cp-0005.ckpt\n",
      "\n",
      "Epoch 00010: saving model to training_2/cp-0010.ckpt\n",
      "\n",
      "Epoch 00015: saving model to training_2/cp-0015.ckpt\n",
      "\n",
      "Epoch 00020: saving model to training_2/cp-0020.ckpt\n",
      "\n",
      "Epoch 00025: saving model to training_2/cp-0025.ckpt\n",
      "\n",
      "Epoch 00030: saving model to training_2/cp-0030.ckpt\n",
      "\n",
      "Epoch 00035: saving model to training_2/cp-0035.ckpt\n",
      "\n",
      "Epoch 00040: saving model to training_2/cp-0040.ckpt\n",
      "\n",
      "Epoch 00045: saving model to training_2/cp-0045.ckpt\n",
      "\n",
      "Epoch 00050: saving model to training_2/cp-0050.ckpt\n",
      "checkpoint\t\t\t  cp-0025.ckpt.index\n",
      "cp-0000.ckpt.data-00000-of-00001  cp-0030.ckpt.data-00000-of-00001\n",
      "cp-0000.ckpt.index\t\t  cp-0030.ckpt.index\n",
      "cp-0005.ckpt.data-00000-of-00001  cp-0035.ckpt.data-00000-of-00001\n",
      "cp-0005.ckpt.index\t\t  cp-0035.ckpt.index\n",
      "cp-0010.ckpt.data-00000-of-00001  cp-0040.ckpt.data-00000-of-00001\n",
      "cp-0010.ckpt.index\t\t  cp-0040.ckpt.index\n",
      "cp-0015.ckpt.data-00000-of-00001  cp-0045.ckpt.data-00000-of-00001\n",
      "cp-0015.ckpt.index\t\t  cp-0045.ckpt.index\n",
      "cp-0020.ckpt.data-00000-of-00001  cp-0050.ckpt.data-00000-of-00001\n",
      "cp-0020.ckpt.index\t\t  cp-0050.ckpt.index\n",
      "cp-0025.ckpt.data-00000-of-00001\n",
      "training_2/cp-0050.ckpt\n"
     ]
    }
   ],
   "source": [
    "# Include the epoch in the file name (uses `str.format`)\n",
    "checkpoint_path = \"training_2/cp-{epoch:04d}.ckpt\"\n",
    "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
    "\n",
    "# Create a callback that saves the model's weights every 5 epochs\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath=checkpoint_path, \n",
    "    verbose=1, \n",
    "    save_weights_only=True,\n",
    "    period=5)\n",
    "\n",
    "# Create a new model instance\n",
    "model = create_model()\n",
    "\n",
    "# Save the weights using the `checkpoint_path` format\n",
    "model.save_weights(checkpoint_path.format(epoch=0))\n",
    "\n",
    "# Train the model with the new callback\n",
    "model.fit(train_images, \n",
    "              train_labels,\n",
    "              epochs=50, \n",
    "              callbacks=[cp_callback],\n",
    "              validation_data=(test_images,test_labels),\n",
    "              verbose=0)\n",
    "\n",
    "# Check the File\n",
    "!ls {checkpoint_dir}\n",
    "\n",
    "# Check Latest Model File\n",
    "latest = tf.train.latest_checkpoint(checkpoint_dir)\n",
    "print(latest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Model\n",
    "가장 마지막까지 Training된 Model을 Load하여 Accuracy를 확인하는 Code이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1 - 0s - loss: 0.5970 - accuracy: 0.8750\n",
      "Restored model, accuracy: 87.50%\n"
     ]
    }
   ],
   "source": [
    "# Create a new model instance\n",
    "model = create_model()\n",
    "\n",
    "# Load the previously saved weights\n",
    "model.load_weights(latest)\n",
    "\n",
    "# Re-evaluate the model\n",
    "loss, acc = model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Manually save weights\n",
    "Keras의 CallBack을 사용하지 않고 저장하는 방법이다.  \n",
    "<code>model.save_weights()</code>로서 저장한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1 - 0s - loss: 0.4418 - accuracy: 0.8670\n",
      "Restored model, accuracy: 86.70%\n"
     ]
    }
   ],
   "source": [
    "# Save the weights\n",
    "path = './checkpoints/my_checkpoint'\n",
    "model.save_weights(path)\n",
    "\n",
    "# Create a new model instance\n",
    "model = create_model()\n",
    "\n",
    "# Restore the weights\n",
    "model.load_weights(path)\n",
    "\n",
    "# Evaluate the model\n",
    "loss,acc = model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print(\"Restored model, accuracy: {:5.2f}%\".format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save and Load the entire model\n",
    "위의 결과인 Check Points는 Model의 Weights들을 저장한 File이다.  \n",
    "따라서 Model에 불러오거나, 적용한 Model에서 평가 및 새롭게 Training이 가능하다.  \n",
    "하지만 Tensorflow Model처럼 Graph의 구조로서 이루워진 것이 아니기 때문에 File자체 만으로는 Model을 만들 수 없다.  \n",
    "\n",
    "따라서 위에서는 다음과 같은 과정을 거쳤다.  \n",
    "```python\n",
    "# Model 선언\n",
    "model = create_model()\n",
    "\n",
    "# Model 가중치 적용\n",
    "model.load_weights(path)\n",
    "```\n",
    "위와 같은 과정이아니라 Model자체를 저장하는 방법에 대해서 알아본다.  \n",
    "Keras에서는 h5 Format을 사용하여 Tensorflow 1.x에서는 .pb 로서 정의하였다.  \n",
    "\n",
    "먼저 Keras에서 제공하는 h5 Format으로서 Model을 정의한다.  \n",
    "이러한 h5 Format으로서 저장하는 것은 다음과 같은 내용을 포함한다.  \n",
    "- The weight values: .ckpt File 같은 Weights 정보\n",
    "- The model's configuratrion: Graph의 정보\n",
    "- The optimizer configuration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 240us/sample - loss: 1.1880 - accuracy: 0.6560\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 53us/sample - loss: 0.4259 - accuracy: 0.8800\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 55us/sample - loss: 0.2868 - accuracy: 0.9310\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 60us/sample - loss: 0.2012 - accuracy: 0.9550\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 58us/sample - loss: 0.1486 - accuracy: 0.9640\n",
      "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter\n",
      "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1\n",
      "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2\n",
      "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay\n",
      "WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate\n",
      "WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.\n",
      "Model: \"sequential_9\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_18 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_19 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "1000/1 - 0s - loss: 0.4504 - accuracy: 0.8680\n",
      "Restored model, accuracy: 86.80%\n"
     ]
    }
   ],
   "source": [
    "# Create and train a new model instance.\n",
    "model = create_model()\n",
    "model.fit(train_images, train_labels, epochs=5)\n",
    "\n",
    "# Save the entire model to a HDF5 file.\n",
    "# The '.h5' extension indicates that the model shuold be saved to HDF5.\n",
    "model.save('my_model.h5')\n",
    "\n",
    "# Recreate the exact same model, including its weights and the optimizer\n",
    "new_model = tf.keras.models.load_model('my_model.h5')\n",
    "\n",
    "# Show the model architecture\n",
    "new_model.summary()\n",
    "\n",
    "loss, acc = new_model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print('Restored model, accuracy: {:5.2f}%'.format(100*acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SaveModel Format\n",
    "위에서는 다음과 같은 내용을 포함한다고 하였습니다.  \n",
    "- The weight values: .ckpt File 같은 Weights 정보\n",
    "- The model's configuratrion: Graph의 정보\n",
    "- The optimizer configuration\n",
    "\n",
    "위의 3가지의 정보를 하나의 .h5 File이아닌 Directory에 나누어서 담는 방법입니다.  \n",
    "먼저 결과부터 살펴보면 Model을 저장하는 Directory의 구조는 다음과 같습니다.  \n",
    "\n",
    "- my_model\n",
    " - assets: Model을 돌리는데 필요한 임의의 파일을 저장합니다. Ex) a vocabulary file used initialize a lookup table.\n",
    " - variables: 모델의 변수\n",
    "   - variables.data\n",
    "   - variables.index\n",
    " - .pb: 모델의 변수 + 구조(전체 그래프)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 238us/sample - loss: 1.1917 - accuracy: 0.6490\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 56us/sample - loss: 0.4432 - accuracy: 0.8740\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 57us/sample - loss: 0.2925 - accuracy: 0.9250\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 52us/sample - loss: 0.2074 - accuracy: 0.9560\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 55us/sample - loss: 0.1540 - accuracy: 0.9670\n",
      "INFO:tensorflow:Assets written to: saved_model/my_model/assets\n",
      "my_model\n",
      "assets\tsaved_model.pb\tvariables\n",
      "Model: \"sequential_10\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_20 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_21 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "1000/1 - 0s - loss: 0.4745 - accuracy: 0.8730\n",
      "Restored model, accuracy: 87.30%\n",
      "(1000, 10)\n"
     ]
    }
   ],
   "source": [
    "# Create and train a new model instance.\n",
    "model = create_model()\n",
    "model.fit(train_images, train_labels, epochs=5)\n",
    "\n",
    "# Save the entire model as a SavedModel.\n",
    "!mkdir -p saved_model\n",
    "model.save('saved_model/my_model') \n",
    "\n",
    "# my_model directory\n",
    "!ls saved_model\n",
    "\n",
    "# Contains an assets folder, saved_model.pb, and variables folder.\n",
    "!ls saved_model/my_model\n",
    "\n",
    "new_model = tf.keras.models.load_model('saved_model/my_model')\n",
    "\n",
    "# Check its architecture\n",
    "new_model.summary()\n",
    "\n",
    "# Evaluate the restored model\n",
    "loss, acc = new_model.evaluate(test_images,  test_labels, verbose=2)\n",
    "print('Restored model, accuracy: {:5.2f}%'.format(100*acc))\n",
    "\n",
    "print(new_model.predict(test_images).shape)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}