notebooks/tensorflow/getting-started-keras.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "qnMpW5Y9nv2l"
   },
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mHF9VCProKJN"
   },
   "source": [
    "# Getting started: Training and prediction with Keras in AI Platform\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/cloud-samples-data/ai-platform/census/keras-tensorflow-cmle.png\" alt=\"Keras, TensorFlow, and AI Platform logos\" width=\"300px\">\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://cloud.google.com/ml-engine/docs/tensorflow/getting-started-keras\">\n",
    "      <img src=\"https://cloud.google.com/_static/images/cloud/icons/favicons/onecloud/super_cloud.png\"\n",
    "           alt=\"Google Cloud logo\" width=\"32px\"> Read on cloud.google.com\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/tensorflow/getting-started-keras.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/tensorflow/getting-started-keras.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hZzRVxNtH-zG"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This tutorial shows how to train a neural network on AI Platform\n",
    "using the Keras sequential API and how to serve predictions from that\n",
    "model.\n",
    "\n",
    "Keras is a high-level API for building and training deep learning models.\n",
    "[tf.keras](https://www.tensorflow.org/guide/keras) is TensorFlow’s\n",
    "implementation of this API.\n",
    "\n",
    "The first two parts of the tutorial walk through training a model on Cloud\n",
    "AI Platform using prewritten Keras code, deploying the trained model to\n",
    "AI Platform, and serving online predictions from the deployed model.\n",
    "\n",
    "The last part of the tutorial digs into the training code used for this model and ensuring it's compatible with AI Platform. To learn more about building\n",
    "machine learning models in Keras more generally, read [TensorFlow's Keras\n",
    "tutorials](https://www.tensorflow.org/tutorials/keras)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iN69d4D9Flrh"
   },
   "source": [
    "### Dataset\n",
    "\n",
    "This tutorial uses the [United States Census Income\n",
    "Dataset](https://archive.ics.uci.edu/ml/datasets/census+income) provided by the\n",
    "[UC Irvine Machine Learning\n",
    "Repository](https://archive.ics.uci.edu/ml/index.php). This dataset contains\n",
    "information about people from a 1994 Census database, including age, education,\n",
    "marital status, occupation, and whether they make more than $50,000 a year."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Su2qu-4CW-YH"
   },
   "source": [
    "### Objective\n",
    "\n",
    "The goal is to train a deep neural network (DNN) using Keras that predicts\n",
    "whether a person makes more than $50,000 a year (target label) based on other\n",
    "Census information about the person (features).\n",
    "\n",
    "This tutorial focuses more on using this model with AI Platform than on\n",
    "the design of the model itself. However, it's always important to think about\n",
    "potential problems and unintended consequences when building machine learning\n",
    "systems. See the [Machine Learning Crash Course exercise about\n",
    "fairness](https://developers.google.com/machine-learning/crash-course/fairness/programming-exercise)\n",
    "to learn about sources of bias in the Census dataset, as well as machine\n",
    "learning fairness more generally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "912RD_3fxGeH"
   },
   "source": [
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
    "\n",
    "* AI Platform\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [AI Platform\n",
    "pricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rgLXkyHEvTVD"
   },
   "source": [
    "## Before you begin\n",
    "\n",
    "You must do several things before you can train and deploy a model in\n",
    "AI Platform:\n",
    "\n",
    "* Set up your local development environment.\n",
    "* Set up a GCP project with billing and the necessary\n",
    "  APIs enabled.\n",
    "* Authenticate your GCP account in this notebook.\n",
    "* Create a Cloud Storage bucket to store your training package and your\n",
    "  trained model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "avDUUQEGTnUo"
   },
   "source": [
    "### Set up your local development environment\n",
    "\n",
    "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
    "all the requirements to run this notebook. You can skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1y4JdKCcTjgJ"
   },
   "source": [
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* Git\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "3. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3.\n",
    "\n",
    "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
    "   Jupyter.\n",
    "\n",
    "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
    "\n",
    "6. Open this notebook in the Jupyter Notebook Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i2qsxysTVc-l"
   },
   "source": [
    "### Set up your GCP project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. [Enable the AI Platform (\"Cloud Machine Learning Engine\") and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
    "\n",
    "4. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "4qxwBA4RM9Lu",
    "outputId": "c4895bd5-2878-4883-b059-443022489bc7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "PROJECT_ID = \"<your-project-id>\" #@param {type:\"string\"}\n",
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TSy-f05IO4LB"
   },
   "source": [
    "### Authenticate your GCP account\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fZQUrHdXNJnk"
   },
   "source": [
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **Machine Learning Engine > AI Platform Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment.\n",
    "\n",
    "6. Enter the path to your service account key as the\n",
    "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W9i6oektpgld",
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your GCP account. This provides access to your\n",
    "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "if 'google.colab' in sys.modules:\n",
    "  from google.colab import auth as google_auth\n",
    "  google_auth.authenticate_user()\n",
    "\n",
    "# If you are running this notebook locally, replace the string below with the\n",
    "# path to your service account key and run this cell to authenticate your GCP\n",
    "# account.\n",
    "else:\n",
    "  %env GOOGLE_APPLICATION_CREDENTIALS ''\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tT061irlJwkg"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you submit a training job using the Cloud SDK, you upload a Python package\n",
    "containing your training code to a Cloud Storage bucket. AI Platform runs\n",
    "the code from this package. In this tutorial, AI Platform also saves the\n",
    "trained model that results from your job in the same bucket. You can then\n",
    "create an AI Platform model version based on this output in order to serve\n",
    "online predictions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets. \n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
    "AI Platform services are\n",
    "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bTxmbDg1I0x1"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"<your-bucket-name>\" #@param {type:\"string\"}\n",
    "REGION = \"us-central1\" #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fsmCk2dwJnLZ"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "160PRO3aJqLD",
    "outputId": "f59db874-8abd-45c2-d562-a4824989fdc6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://<your-bucket-name>/...\n"
     ]
    }
   ],
   "source": [
    "! gsutil mb -l $REGION gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iTc8RvKlSjIG"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OsB4T3sbSb2d"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aRVMEU2Qshm4"
   },
   "source": [
    "## Part 1. Quickstart for training in AI Platform\n",
    "\n",
    "This section of the tutorial walks you through submitting a training job to Cloud\n",
    "AI Platform. This job runs sample code that uses Keras to train a deep neural\n",
    "network on the United States Census data. It outputs the trained model as a\n",
    "[TensorFlow SavedModel\n",
    "directory](https://www.tensorflow.org/guide/saved_model#save_and_restore_models)\n",
    "in your Cloud Storage bucket.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8zr6lj66UlMn"
   },
   "source": [
    "### Get training code and dependencies\n",
    "\n",
    "First, download the training code and change the notebook's working directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "Icz22E69smnD",
    "outputId": "a5113351-142c-4c0f-e640-5cccea37a481"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'cloudml-samples'...\n",
      "remote: Enumerating objects: 404, done.\u001b[K\n",
      "remote: Counting objects: 100% (404/404), done.\u001b[K\n",
      "remote: Compressing objects: 100% (333/333), done.\u001b[K\n",
      "remote: Total 404 (delta 110), reused 199 (delta 46), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (404/404), 22.67 MiB | 19.76 MiB/s, done.\n",
      "Resolving deltas: 100% (110/110), done.\n",
      "/content/cloudml-samples/census/tf-keras\n"
     ]
    }
   ],
   "source": [
    "# Clone the repository of AI Platform samples\n",
    "! git clone --depth 1 https://github.com/GoogleCloudPlatform/cloudml-samples\n",
    "\n",
    "# Set the working directory to the sample code directory\n",
    "%cd cloudml-samples/census/tf-keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MhubJDDXSVv3"
   },
   "source": [
    "Notice that the training code is structured as a Python package in the\n",
    "`trainer/` subdirectory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "uZ_nfuPJlNpi",
    "outputId": "766d898d-92cf-4560-be1e-3d1cc35b2a0d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".:\n",
      "README.md  requirements.txt  trainer/\n",
      "\n",
      "./trainer:\n",
      "__init__.py  model.py  task.py\tutil.py\n"
     ]
    }
   ],
   "source": [
    "# `ls` shows the working directory's contents. The `p` flag adds trailing \n",
    "# slashes to subdirectory names. The `R` flag lists subdirectories recursively.\n",
    "! ls -pR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7KA87o4HUhby"
   },
   "source": [
    "Run the following cell to install Python dependencies needed to train the model locally. When you run the training job in AI Platform,\n",
    "dependencies are preinstalled based on the [runtime\n",
    "version](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list)\n",
    "you choose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "id": "Wm5w1UrmVU7O",
    "outputId": "ed03f67c-a605-4dcd-e17e-5dfd70426641"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 1)) (1.17.4)\n",
      "Requirement already satisfied: pandas>=0.22 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 2)) (0.25.3)\n",
      "Requirement already satisfied: six>=1.11 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 3)) (1.13.0)\n",
      "Requirement already satisfied: tensorflow<2,>=1.15 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 4)) (1.15.0)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.22->-r requirements.txt (line 2)) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.22->-r requirements.txt (line 2)) (2019.3)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.1.0)\n",
      "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.2.2)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.8.1)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (3.11.1)\n",
      "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.15.1)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.11.2)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/lib/python3/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.29.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (3.1.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.25.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.8.1)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.1.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.0.8)\n",
      "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (1.15.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.1.8)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.6.1->tensorflow<2,>=1.15->-r requirements.txt (line 4)) (42.0.2)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.8->tensorflow<2,>=1.15->-r requirements.txt (line 4)) (2.10.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2,>=1.15->-r requirements.txt (line 4)) (3.1.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2,>=1.15->-r requirements.txt (line 4)) (0.16.0)\n"
     ]
    }
   ],
   "source": [
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iSrzwuchvcgv"
   },
   "source": [
    "### Train your model locally\n",
    "\n",
    "Before training on AI Platform, train the job locally to verify the file\n",
    "structure and packaging is correct.\n",
    "\n",
    "For a complex or resource-intensive job, you\n",
    "may want to train locally on a small sample of your dataset to verify your code.\n",
    "Then you can run the job on AI Platform to train on the whole dataset.\n",
    "\n",
    "This sample runs a relatively quick job on a small dataset, so the local\n",
    "training and the AI Platform job run the same code on the same data.\n",
    "\n",
    "Run the following cell to train a model locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1890
    },
    "colab_type": "code",
    "id": "D5PIljnYveDN",
    "outputId": "04cd1942-6e37-403d-c298-2d1d65a9b30f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [ml_engine/local_python].\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "2019-03-27 17:53:24.297156: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2019-03-27 17:53:24.297428: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2441b80 executing computations on platform Host. Devices:\n",
      "2019-03-27 17:53:24.297464: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "\n",
      "Epoch 00001: LearningRateScheduler reducing learning rate to 0.02.\n",
      "Epoch 1/20\n",
      "254/254 [==============================] - 1s 5ms/step - loss: 0.5032 - acc: 0.7890 - val_loss: 0.4553 - val_acc: 0.8030\n",
      "\n",
      "Epoch 00002: LearningRateScheduler reducing learning rate to 0.015.\n",
      "Epoch 2/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3616 - acc: 0.8362 - val_loss: 0.3273 - val_acc: 0.8468\n",
      "\n",
      "Epoch 00003: LearningRateScheduler reducing learning rate to 0.0125.\n",
      "Epoch 3/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3411 - acc: 0.8450 - val_loss: 0.3294 - val_acc: 0.8447\n",
      "\n",
      "Epoch 00004: LearningRateScheduler reducing learning rate to 0.01125.\n",
      "Epoch 4/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3362 - acc: 0.8454 - val_loss: 0.3566 - val_acc: 0.8410\n",
      "\n",
      "Epoch 00005: LearningRateScheduler reducing learning rate to 0.010625.\n",
      "Epoch 5/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3325 - acc: 0.8469 - val_loss: 0.3264 - val_acc: 0.8507\n",
      "\n",
      "Epoch 00006: LearningRateScheduler reducing learning rate to 0.0103125.\n",
      "Epoch 6/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3316 - acc: 0.8471 - val_loss: 0.3228 - val_acc: 0.8515\n",
      "\n",
      "Epoch 00007: LearningRateScheduler reducing learning rate to 0.01015625.\n",
      "Epoch 7/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3297 - acc: 0.8475 - val_loss: 0.3308 - val_acc: 0.8452\n",
      "\n",
      "Epoch 00008: LearningRateScheduler reducing learning rate to 0.010078125.\n",
      "Epoch 8/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3287 - acc: 0.8479 - val_loss: 0.3449 - val_acc: 0.8394\n",
      "\n",
      "Epoch 00009: LearningRateScheduler reducing learning rate to 0.0100390625.\n",
      "Epoch 9/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3274 - acc: 0.8488 - val_loss: 0.3213 - val_acc: 0.8522\n",
      "\n",
      "Epoch 00010: LearningRateScheduler reducing learning rate to 0.01001953125.\n",
      "Epoch 10/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3256 - acc: 0.8488 - val_loss: 0.3302 - val_acc: 0.8508\n",
      "\n",
      "Epoch 00011: LearningRateScheduler reducing learning rate to 0.010009765625.\n",
      "Epoch 11/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3280 - acc: 0.8502 - val_loss: 0.3342 - val_acc: 0.8443\n",
      "\n",
      "Epoch 00012: LearningRateScheduler reducing learning rate to 0.010004882812500001.\n",
      "Epoch 12/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3283 - acc: 0.8502 - val_loss: 0.3511 - val_acc: 0.8506\n",
      "\n",
      "Epoch 00013: LearningRateScheduler reducing learning rate to 0.01000244140625.\n",
      "Epoch 13/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3258 - acc: 0.8498 - val_loss: 0.3450 - val_acc: 0.8491\n",
      "\n",
      "Epoch 00014: LearningRateScheduler reducing learning rate to 0.010001220703125.\n",
      "Epoch 14/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3272 - acc: 0.8500 - val_loss: 0.3220 - val_acc: 0.8524\n",
      "\n",
      "Epoch 00015: LearningRateScheduler reducing learning rate to 0.0100006103515625.\n",
      "Epoch 15/20\n",
      "254/254 [==============================] - 1s 3ms/step - loss: 0.3261 - acc: 0.8498 - val_loss: 0.3221 - val_acc: 0.8520\n",
      "\n",
      "Epoch 00016: LearningRateScheduler reducing learning rate to 0.01000030517578125.\n",
      "Epoch 16/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3239 - acc: 0.8509 - val_loss: 0.3232 - val_acc: 0.8480\n",
      "\n",
      "Epoch 00017: LearningRateScheduler reducing learning rate to 0.010000152587890625.\n",
      "Epoch 17/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3275 - acc: 0.8484 - val_loss: 0.3377 - val_acc: 0.8540\n",
      "\n",
      "Epoch 00018: LearningRateScheduler reducing learning rate to 0.010000076293945313.\n",
      "Epoch 18/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3242 - acc: 0.8490 - val_loss: 0.3275 - val_acc: 0.8522\n",
      "\n",
      "Epoch 00019: LearningRateScheduler reducing learning rate to 0.010000038146972657.\n",
      "Epoch 19/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3236 - acc: 0.8507 - val_loss: 0.3302 - val_acc: 0.8471\n",
      "\n",
      "Epoch 00020: LearningRateScheduler reducing learning rate to 0.010000019073486329.\n",
      "Epoch 20/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3259 - acc: 0.8490 - val_loss: 0.3218 - val_acc: 0.8518\n",
      "WARNING:tensorflow:This model was compiled with a Keras optimizer (<tensorflow.python.keras.optimizers.RMSprop object at 0x7fa74da22278>) but is being saved in TensorFlow format with `save_weights`. The model's weights will be saved, but unlike with TensorFlow optimizers in the TensorFlow format the optimizer's state will not be saved.\n",
      "\n",
      "Consider using a TensorFlow optimizer from `tf.train`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/network.py:1436: update_checkpoint_state (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.train.CheckpointManager to manage checkpoints rather than manually editing the Checkpoint proto.\n",
      "WARNING:tensorflow:Model was compiled with an optimizer, but the optimizer is not from `tf.train` (e.g. `tf.train.AdagradOptimizer`). Only the serving graph was exported. The train and evaluate graphs were not added to the SavedModel.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "INFO:tensorflow:No assets to save.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: local-training-output/keras_export/1553709223/saved_model.pb\n",
      "Model exported to:  local-training-output/keras_export/1553709223\n"
     ]
    }
   ],
   "source": [
    "# Explicitly tell `gcloud ai-platform local train` to use Python 3 \n",
    "! gcloud config set ml_engine/local_python $(which python3)\n",
    "\n",
    "# This is similar to `python -m trainer.task --job-dir local-training-output`\n",
    "# but it better replicates the AI Platform environment, especially for\n",
    "# distributed training (not applicable here).\n",
    "! gcloud ai-platform local train \\\n",
    "  --package-path trainer \\\n",
    "  --module-name trainer.task \\\n",
    "  --job-dir local-training-output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rAX4hZip53SR"
   },
   "source": [
    "### Train your model using AI Platform\n",
    "\n",
    "Next, submit a training job to AI Platform. This runs the training module\n",
    "in the cloud and exports the trained model to Cloud Storage.\n",
    "\n",
    "First, give your training job a name and choose a directory within your Cloud\n",
    "Storage bucket for saving intermediate and output files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "05lDch9-0-2v"
   },
   "outputs": [],
   "source": [
    "JOB_NAME = 'my_first_keras_job'\n",
    "JOB_DIR = 'gs://' + BUCKET_NAME + '/keras-job-dir'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yp9nyrZ01a2q"
   },
   "source": [
    "Run the following command to package the `trainer/` directory, upload it to the\n",
    "specified `--job-dir`, and instruct AI Platform to run the\n",
    "`trainer.task` module from that package.\n",
    "\n",
    "The `--stream-logs` flag lets you view training logs in the cell below. You can\n",
    "also see logs and other job details in the GCP Console."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J2WGqwAzc3xM"
   },
   "source": [
    "### Hyperparameter tuning\n",
    "\n",
    "You can optionally perform hyperparameter tuning by using the included\n",
    "`hptuning_config.yaml` configuration file. This file tells AI Platform to tune the batch size and learning rate for training over multiple trials to maximize accuracy.\n",
    "\n",
    "In this example, the training code uses a [TensorBoard\n",
    "callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard),\n",
    "which [creates TensorFlow `Summary`\n",
    "`Event`s](https://www.tensorflow.org/api_docs/python/tf/summary/FileWriter#add_summary)\n",
    "during training. AI Platform uses these events to track the metric you want to\n",
    "optimize. Learn more about [hyperparameter tuning in\n",
    "AI Platform Training](https://cloud.google.com/ml-engine/docs/tensorflow/hyperparameter-tuning-overview)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 7500
    },
    "colab_type": "code",
    "id": "1haRe54v53CN",
    "outputId": "dfa15c82-604e-4451-fa19-0dc13ced3ced"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job [my_first_keras_job] submitted successfully.\n",
      "INFO\t2019-03-27 17:54:27 +0000\tservice\t\tValidating job requirements...\n",
      "INFO\t2019-03-27 17:54:27 +0000\tservice\t\tJob creation request has been successfully validated.\n",
      "INFO\t2019-03-27 17:54:27 +0000\tservice\t\tJob my_first_keras_job is queued.\n",
      "INFO\t2019-03-27 17:54:27 +0000\tservice\t\tWaiting for job to be provisioned.\n",
      "INFO\t2019-03-27 17:54:30 +0000\tservice\t\tWaiting for training program to start.\n",
      "INFO\t2019-03-27 17:56:09 +0000\tmaster-replica-0\t\tRunning task with arguments: --cluster={\"master\": [\"127.0.0.1:2222\"]} --task={\"type\": \"master\", \"index\": 0} --job={  \"package_uris\": [\"gs://<your-bucket-name>/keras-job-dir/packages/dcc159f40836cff74a27866227b327b0a8ccb5266194e76cff5368266b6d1cdd/trainer-0.0.0.tar.gz\"],  \"python_module\": \"trainer.task\",  \"region\": \"us-central1\",  \"runtime_version\": \"1.15\",  \"job_dir\": \"gs://<your-bucket-name>/keras-job-dir\",  \"run_on_raw_vm\": true,  \"python_version\": \"3.7\"}\n",
      "WARNING\t2019-03-27 17:56:09 +0000\tmaster-replica-0\t\tFrom /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "WARNING\t2019-03-27 17:56:09 +0000\tmaster-replica-0\t\tInstructions for updating:\n",
      "WARNING\t2019-03-27 17:56:09 +0000\tmaster-replica-0\t\tColocations handled automatically by placer.\n",
      "INFO\t2019-03-27 17:56:18 +0000\tmaster-replica-0\t\tRunning module trainer.task.\n",
      "INFO\t2019-03-27 17:56:18 +0000\tmaster-replica-0\t\tDownloading the package: gs://<your-bucket-name>/keras-job-dir/packages/dcc159f40836cff74a27866227b327b0a8ccb5266194e76cff5368266b6d1cdd/trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:18 +0000\tmaster-replica-0\t\tRunning command: gsutil -q cp gs://<your-bucket-name>/keras-job-dir/packages/dcc159f40836cff74a27866227b327b0a8ccb5266194e76cff5368266b6d1cdd/trainer-0.0.0.tar.gz trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:20 +0000\tmaster-replica-0\t\tInstalling the package: gs://<your-bucket-name>/keras-job-dir/packages/dcc159f40836cff74a27866227b327b0a8ccb5266194e76cff5368266b6d1cdd/trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:20 +0000\tmaster-replica-0\t\tRunning command: pip3 install --user --upgrade --force-reinstall --no-deps trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:22 +0000\tmaster-replica-0\t\tProcessing ./trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:22 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt\n",
      "INFO\t2019-03-27 17:56:22 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tBuilding wheels for collected packages: trainer\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\t  Building wheel for trainer (setup.py): started\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\t  Building wheel for trainer (setup.py): finished with status 'done'\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\t  Stored in directory: /root/.cache/pip/wheels/0d/1b/db/f8e86b296734f0b137e17e5d34862f4ae4faf8388755c6272f\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tSuccessfully built trainer\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tInstalling collected packages: trainer\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tSuccessfully installed trainer-0.0.0\n",
      "INFO\t2019-03-27 17:56:23 +0000\tmaster-replica-0\t\tRunning command: pip3 install --user trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:24 +0000\tmaster-replica-0\t\tProcessing ./trainer-0.0.0.tar.gz\n",
      "INFO\t2019-03-27 17:56:24 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt\n",
      "INFO\t2019-03-27 17:56:24 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt\n",
      "INFO\t2019-03-27 17:56:24 +0000\tmaster-replica-0\t\tBuilding wheels for collected packages: trainer\n",
      "INFO\t2019-03-27 17:56:24 +0000\tmaster-replica-0\t\t  Building wheel for trainer (setup.py): started\n",
      "INFO\t2019-03-27 17:56:25 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt\n",
      "INFO\t2019-03-27 17:56:25 +0000\tmaster-replica-0\t\tGenerating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt\n",
      "INFO\t2019-03-27 17:56:25 +0000\tmaster-replica-0\t\t  Building wheel for trainer (setup.py): finished with status 'done'\n",
      "INFO\t2019-03-27 17:56:25 +0000\tmaster-replica-0\t\t  Stored in directory: /root/.cache/pip/wheels/0d/1b/db/f8e86b296734f0b137e17e5d34862f4ae4faf8388755c6272f\n",
      "INFO\t2019-03-27 17:56:25 +0000\tmaster-replica-0\t\tSuccessfully built trainer\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\tInstalling collected packages: trainer\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\t  Found existing installation: trainer 0.0.0\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\t    Uninstalling trainer-0.0.0:\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\t      Successfully uninstalled trainer-0.0.0\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\tSuccessfully installed trainer-0.0.0\n",
      "INFO\t2019-03-27 17:56:29 +0000\tmaster-replica-0\t\tRunning command: python3 -m trainer.task --job-dir gs://<your-bucket-name>/keras-job-dir\n",
      "WARNING\t2019-03-27 17:56:43 +0000\tmaster-replica-0\t\tFrom /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "WARNING\t2019-03-27 17:56:43 +0000\tmaster-replica-0\t\tInstructions for updating:\n",
      "WARNING\t2019-03-27 17:56:43 +0000\tmaster-replica-0\t\tColocations handled automatically by placer.\n",
      "INFO\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tYour CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "INFO\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tCPU Frequency: 2300000000 Hz\n",
      "INFO\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tXLA service 0x4f15c40 executing computations on platform Host. Devices:\n",
      "INFO\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\t  StreamExecutor device (0): <undefined>, <undefined>\n",
      "WARNING\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tFrom /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "WARNING\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tInstructions for updating:\n",
      "WARNING\t2019-03-27 17:56:44 +0000\tmaster-replica-0\t\tUse tf.cast instead.\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\tEpoch 00001: LearningRateScheduler reducing learning rate to 0.02.\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\tEpoch 1/20\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1:06 - loss: 0.5855 - acc: 0.789\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 16/254 [>.............................] - ETA: 4s - loss: 3.8615 - acc: 0.7495 \n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 34/254 [===>..........................] - ETA: 2s - loss: 3.1560 - acc: 0.741\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 52/254 [=====>........................] - ETA: 1s - loss: 2.2601 - acc: 0.751\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 69/254 [=======>......................] - ETA: 1s - loss: 1.8414 - acc: 0.750\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 87/254 [=========>....................] - ETA: 0s - loss: 1.5580 - acc: 0.754\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t105/254 [===========>..................] - ETA: 0s - loss: 1.3674 - acc: 0.761\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t121/254 [=============>................] - ETA: 0s - loss: 1.2418 - acc: 0.767\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t141/254 [===============>..............] - ETA: 0s - loss: 1.1292 - acc: 0.770\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t161/254 [==================>...........] - ETA: 0s - loss: 1.0420 - acc: 0.774\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t179/254 [====================>.........] - ETA: 0s - loss: 0.9786 - acc: 0.778\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t198/254 [======================>.......] - ETA: 0s - loss: 0.9222 - acc: 0.783\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t218/254 [========================>.....] - ETA: 0s - loss: 0.8751 - acc: 0.786\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t238/254 [===========================>..] - ETA: 0s - loss: 0.8347 - acc: 0.788\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.8076 - acc: 0.7896 - val_loss: 0.4046 - val_acc: 0.8322\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\tEpoch 00002: LearningRateScheduler reducing learning rate to 0.015.\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\tEpoch 2/20\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3897 - acc: 0.843\n",
      "INFO\t2019-03-27 17:56:45 +0000\tmaster-replica-0\t\t 21/254 [=>............................] - ETA: 0s - loss: 0.3849 - acc: 0.830\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 40/254 [===>..........................] - ETA: 0s - loss: 0.3795 - acc: 0.827\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 61/254 [======>.......................] - ETA: 0s - loss: 0.3706 - acc: 0.826\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 80/254 [========>.....................] - ETA: 0s - loss: 0.3666 - acc: 0.826\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 99/254 [==========>...................] - ETA: 0s - loss: 0.3618 - acc: 0.829\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t119/254 [=============>................] - ETA: 0s - loss: 0.3604 - acc: 0.830\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t139/254 [===============>..............] - ETA: 0s - loss: 0.3622 - acc: 0.829\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t158/254 [=================>............] - ETA: 0s - loss: 0.3593 - acc: 0.831\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t177/254 [===================>..........] - ETA: 0s - loss: 0.3601 - acc: 0.831\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t197/254 [======================>.......] - ETA: 0s - loss: 0.3598 - acc: 0.830\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t216/254 [========================>.....] - ETA: 0s - loss: 0.3599 - acc: 0.831\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t237/254 [==========================>...] - ETA: 0s - loss: 0.3597 - acc: 0.831\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3580 - acc: 0.8321 - val_loss: 0.3400 - val_acc: 0.8372\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\tEpoch 00003: LearningRateScheduler reducing learning rate to 0.0125.\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\tEpoch 3/20\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1s - loss: 0.3455 - acc: 0.820\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 17/254 [=>............................] - ETA: 0s - loss: 0.3449 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 36/254 [===>..........................] - ETA: 0s - loss: 0.3414 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 55/254 [=====>........................] - ETA: 0s - loss: 0.3324 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 74/254 [=======>......................] - ETA: 0s - loss: 0.3378 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t 91/254 [=========>....................] - ETA: 0s - loss: 0.3372 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:46 +0000\tmaster-replica-0\t\t107/254 [===========>..................] - ETA: 0s - loss: 0.3394 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t122/254 [=============>................] - ETA: 0s - loss: 0.3433 - acc: 0.839\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t138/254 [===============>..............] - ETA: 0s - loss: 0.3426 - acc: 0.839\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t158/254 [=================>............] - ETA: 0s - loss: 0.3421 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t175/254 [===================>..........] - ETA: 0s - loss: 0.3427 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t192/254 [=====================>........] - ETA: 0s - loss: 0.3417 - acc: 0.841\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t209/254 [=======================>......] - ETA: 0s - loss: 0.3405 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t226/254 [=========================>....] - ETA: 0s - loss: 0.3409 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t247/254 [============================>.] - ETA: 0s - loss: 0.3412 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3409 - acc: 0.8425 - val_loss: 0.3308 - val_acc: 0.8496\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\tEpoch 00004: LearningRateScheduler reducing learning rate to 0.01125.\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\tEpoch 4/20\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.4057 - acc: 0.820\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t 17/254 [=>............................] - ETA: 0s - loss: 0.3522 - acc: 0.832\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t 34/254 [===>..........................] - ETA: 0s - loss: 0.3467 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t 54/254 [=====>........................] - ETA: 0s - loss: 0.3441 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t 71/254 [=======>......................] - ETA: 0s - loss: 0.3448 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t 91/254 [=========>....................] - ETA: 0s - loss: 0.3435 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t111/254 [============>.................] - ETA: 0s - loss: 0.3417 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t130/254 [==============>...............] - ETA: 0s - loss: 0.3392 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t149/254 [================>.............] - ETA: 0s - loss: 0.3386 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t167/254 [==================>...........] - ETA: 0s - loss: 0.3370 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:47 +0000\tmaster-replica-0\t\t188/254 [=====================>........] - ETA: 0s - loss: 0.3361 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t208/254 [=======================>......] - ETA: 0s - loss: 0.3349 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t228/254 [=========================>....] - ETA: 0s - loss: 0.3364 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t245/254 [===========================>..] - ETA: 0s - loss: 0.3367 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3366 - acc: 0.8451 - val_loss: 0.3431 - val_acc: 0.8319\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\tEpoch 00005: LearningRateScheduler reducing learning rate to 0.010625.\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\tEpoch 5/20\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3805 - acc: 0.828\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 22/254 [=>............................] - ETA: 0s - loss: 0.3307 - acc: 0.854\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 41/254 [===>..........................] - ETA: 0s - loss: 0.3297 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 59/254 [=====>........................] - ETA: 0s - loss: 0.3334 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 74/254 [=======>......................] - ETA: 0s - loss: 0.3299 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 87/254 [=========>....................] - ETA: 0s - loss: 0.3323 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t 97/254 [==========>...................] - ETA: 0s - loss: 0.3336 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t107/254 [===========>..................] - ETA: 0s - loss: 0.3344 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t120/254 [=============>................] - ETA: 0s - loss: 0.3367 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t136/254 [===============>..............] - ETA: 0s - loss: 0.3368 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t151/254 [================>.............] - ETA: 0s - loss: 0.3359 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t169/254 [==================>...........] - ETA: 0s - loss: 0.3373 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t187/254 [=====================>........] - ETA: 0s - loss: 0.3364 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t205/254 [=======================>......] - ETA: 0s - loss: 0.3348 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:48 +0000\tmaster-replica-0\t\t223/254 [=========================>....] - ETA: 0s - loss: 0.3345 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3353 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3339 - acc: 0.8453 - val_loss: 0.3486 - val_acc: 0.8504\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\tEpoch 00006: LearningRateScheduler reducing learning rate to 0.0103125.\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\tEpoch 6/20\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.5068 - acc: 0.789\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 18/254 [=>............................] - ETA: 0s - loss: 0.3490 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 38/254 [===>..........................] - ETA: 0s - loss: 0.3355 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 58/254 [=====>........................] - ETA: 0s - loss: 0.3312 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 78/254 [========>.....................] - ETA: 0s - loss: 0.3311 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 99/254 [==========>...................] - ETA: 0s - loss: 0.3317 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t116/254 [============>.................] - ETA: 0s - loss: 0.3312 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t134/254 [==============>...............] - ETA: 0s - loss: 0.3326 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t153/254 [=================>............] - ETA: 0s - loss: 0.3326 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t169/254 [==================>...........] - ETA: 0s - loss: 0.3351 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t187/254 [=====================>........] - ETA: 0s - loss: 0.3351 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t204/254 [=======================>......] - ETA: 0s - loss: 0.3346 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t220/254 [========================>.....] - ETA: 0s - loss: 0.3345 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t237/254 [==========================>...] - ETA: 0s - loss: 0.3330 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3317 - acc: 0.8483 - val_loss: 0.3297 - val_acc: 0.8402\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\tEpoch 00007: LearningRateScheduler reducing learning rate to 0.01015625.\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\tEpoch 7/20\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.2934 - acc: 0.890\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 19/254 [=>............................] - ETA: 0s - loss: 0.3127 - acc: 0.858\n",
      "INFO\t2019-03-27 17:56:49 +0000\tmaster-replica-0\t\t 37/254 [===>..........................] - ETA: 0s - loss: 0.3242 - acc: 0.855\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 55/254 [=====>........................] - ETA: 0s - loss: 0.3300 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 74/254 [=======>......................] - ETA: 0s - loss: 0.3345 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 93/254 [=========>....................] - ETA: 0s - loss: 0.3329 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t111/254 [============>.................] - ETA: 0s - loss: 0.3327 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t128/254 [==============>...............] - ETA: 0s - loss: 0.3327 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t147/254 [================>.............] - ETA: 0s - loss: 0.3307 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t163/254 [==================>...........] - ETA: 0s - loss: 0.3310 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t183/254 [====================>.........] - ETA: 0s - loss: 0.3316 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t202/254 [======================>.......] - ETA: 0s - loss: 0.3335 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t219/254 [========================>.....] - ETA: 0s - loss: 0.3312 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t238/254 [===========================>..] - ETA: 0s - loss: 0.3312 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3316 - acc: 0.8480 - val_loss: 0.3250 - val_acc: 0.8461\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\tEpoch 00008: LearningRateScheduler reducing learning rate to 0.010078125.\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\tEpoch 8/20\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3214 - acc: 0.859\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 19/254 [=>............................] - ETA: 0s - loss: 0.3520 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 34/254 [===>..........................] - ETA: 0s - loss: 0.3456 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 49/254 [====>.........................] - ETA: 0s - loss: 0.3347 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 64/254 [======>.......................] - ETA: 0s - loss: 0.3296 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 79/254 [========>.....................] - ETA: 0s - loss: 0.3293 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t 95/254 [==========>...................] - ETA: 0s - loss: 0.3288 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:50 +0000\tmaster-replica-0\t\t112/254 [============>.................] - ETA: 0s - loss: 0.3285 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t128/254 [==============>...............] - ETA: 0s - loss: 0.3307 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t144/254 [================>.............] - ETA: 0s - loss: 0.3294 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t161/254 [==================>...........] - ETA: 0s - loss: 0.3294 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t175/254 [===================>..........] - ETA: 0s - loss: 0.3289 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t190/254 [=====================>........] - ETA: 0s - loss: 0.3308 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t206/254 [=======================>......] - ETA: 0s - loss: 0.3302 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t221/254 [=========================>....] - ETA: 0s - loss: 0.3310 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t237/254 [==========================>...] - ETA: 0s - loss: 0.3327 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3313 - acc: 0.8483 - val_loss: 0.3255 - val_acc: 0.8500\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\tEpoch 00009: LearningRateScheduler reducing learning rate to 0.0100390625.\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\tEpoch 9/20\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.4250 - acc: 0.796\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t 20/254 [=>............................] - ETA: 0s - loss: 0.3384 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t 35/254 [===>..........................] - ETA: 0s - loss: 0.3326 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t 51/254 [=====>........................] - ETA: 0s - loss: 0.3307 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t 68/254 [=======>......................] - ETA: 0s - loss: 0.3312 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t 85/254 [=========>....................] - ETA: 0s - loss: 0.3337 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t102/254 [===========>..................] - ETA: 0s - loss: 0.3346 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t118/254 [============>.................] - ETA: 0s - loss: 0.3358 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t134/254 [==============>...............] - ETA: 0s - loss: 0.3320 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:51 +0000\tmaster-replica-0\t\t150/254 [================>.............] - ETA: 0s - loss: 0.3319 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t168/254 [==================>...........] - ETA: 0s - loss: 0.3337 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t187/254 [=====================>........] - ETA: 0s - loss: 0.3332 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t206/254 [=======================>......] - ETA: 0s - loss: 0.3315 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t224/254 [=========================>....] - ETA: 0s - loss: 0.3311 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3304 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3314 - acc: 0.8485 - val_loss: 0.3236 - val_acc: 0.8520\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\tEpoch 00010: LearningRateScheduler reducing learning rate to 0.01001953125.\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\tEpoch 10/20\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.2766 - acc: 0.890\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 18/254 [=>............................] - ETA: 0s - loss: 0.3305 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 33/254 [==>...........................] - ETA: 0s - loss: 0.3337 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 48/254 [====>.........................] - ETA: 0s - loss: 0.3414 - acc: 0.843\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 62/254 [======>.......................] - ETA: 0s - loss: 0.3370 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 73/254 [=======>......................] - ETA: 0s - loss: 0.3332 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 84/254 [========>.....................] - ETA: 0s - loss: 0.3283 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t 94/254 [==========>...................] - ETA: 0s - loss: 0.3290 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t105/254 [===========>..................] - ETA: 0s - loss: 0.3292 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t119/254 [=============>................] - ETA: 0s - loss: 0.3262 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t133/254 [==============>...............] - ETA: 0s - loss: 0.3295 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t148/254 [================>.............] - ETA: 0s - loss: 0.3296 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:52 +0000\tmaster-replica-0\t\t164/254 [==================>...........] - ETA: 0s - loss: 0.3290 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t181/254 [====================>.........] - ETA: 0s - loss: 0.3303 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t199/254 [======================>.......] - ETA: 0s - loss: 0.3291 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t217/254 [========================>.....] - ETA: 0s - loss: 0.3297 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t236/254 [==========================>...] - ETA: 0s - loss: 0.3295 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.3291 - acc: 0.8494 - val_loss: 0.3264 - val_acc: 0.8516\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\tEpoch 00011: LearningRateScheduler reducing learning rate to 0.010009765625.\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\tEpoch 11/20\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3474 - acc: 0.859\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t 18/254 [=>............................] - ETA: 0s - loss: 0.3394 - acc: 0.857\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t 37/254 [===>..........................] - ETA: 0s - loss: 0.3287 - acc: 0.855\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t 56/254 [=====>........................] - ETA: 0s - loss: 0.3220 - acc: 0.855\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t 74/254 [=======>......................] - ETA: 0s - loss: 0.3257 - acc: 0.854\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t 92/254 [=========>....................] - ETA: 0s - loss: 0.3259 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t111/254 [============>.................] - ETA: 0s - loss: 0.3267 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t130/254 [==============>...............] - ETA: 0s - loss: 0.3266 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t147/254 [================>.............] - ETA: 0s - loss: 0.3260 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t166/254 [==================>...........] - ETA: 0s - loss: 0.3253 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t185/254 [====================>.........] - ETA: 0s - loss: 0.3249 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t204/254 [=======================>......] - ETA: 0s - loss: 0.3257 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t221/254 [=========================>....] - ETA: 0s - loss: 0.3275 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t241/254 [===========================>..] - ETA: 0s - loss: 0.3264 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3270 - acc: 0.8494 - val_loss: 0.3246 - val_acc: 0.8499\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\tEpoch 00012: LearningRateScheduler reducing learning rate to 0.010004882812500001.\n",
      "INFO\t2019-03-27 17:56:53 +0000\tmaster-replica-0\t\tEpoch 12/20\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3179 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 19/254 [=>............................] - ETA: 0s - loss: 0.3353 - acc: 0.843\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 39/254 [===>..........................] - ETA: 0s - loss: 0.3319 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 59/254 [=====>........................] - ETA: 0s - loss: 0.3268 - acc: 0.854\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 79/254 [========>.....................] - ETA: 0s - loss: 0.3309 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 99/254 [==========>...................] - ETA: 0s - loss: 0.3315 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t119/254 [=============>................] - ETA: 0s - loss: 0.3311 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t135/254 [==============>...............] - ETA: 0s - loss: 0.3284 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t153/254 [=================>............] - ETA: 0s - loss: 0.3270 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t172/254 [===================>..........] - ETA: 0s - loss: 0.3263 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t190/254 [=====================>........] - ETA: 0s - loss: 0.3263 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t206/254 [=======================>......] - ETA: 0s - loss: 0.3271 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t225/254 [=========================>....] - ETA: 0s - loss: 0.3274 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3271 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3276 - acc: 0.8500 - val_loss: 0.3452 - val_acc: 0.8444\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\tEpoch 00013: LearningRateScheduler reducing learning rate to 0.01000244140625.\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\tEpoch 13/20\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3248 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 19/254 [=>............................] - ETA: 0s - loss: 0.3278 - acc: 0.837\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 38/254 [===>..........................] - ETA: 0s - loss: 0.3317 - acc: 0.842\n",
      "INFO\t2019-03-27 17:56:54 +0000\tmaster-replica-0\t\t 53/254 [=====>........................] - ETA: 0s - loss: 0.3286 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 71/254 [=======>......................] - ETA: 0s - loss: 0.3204 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 88/254 [=========>....................] - ETA: 0s - loss: 0.3232 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t105/254 [===========>..................] - ETA: 0s - loss: 0.3271 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t124/254 [=============>................] - ETA: 0s - loss: 0.3308 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t141/254 [===============>..............] - ETA: 0s - loss: 0.3316 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t157/254 [=================>............] - ETA: 0s - loss: 0.3294 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t171/254 [===================>..........] - ETA: 0s - loss: 0.3298 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t189/254 [=====================>........] - ETA: 0s - loss: 0.3304 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t206/254 [=======================>......] - ETA: 0s - loss: 0.3295 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t225/254 [=========================>....] - ETA: 0s - loss: 0.3315 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3305 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3301 - acc: 0.8485 - val_loss: 0.3439 - val_acc: 0.8439\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\tEpoch 00014: LearningRateScheduler reducing learning rate to 0.010001220703125.\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\tEpoch 14/20\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1s - loss: 0.3098 - acc: 0.890\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 16/254 [>.............................] - ETA: 0s - loss: 0.3104 - acc: 0.855\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 33/254 [==>...........................] - ETA: 0s - loss: 0.3184 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 49/254 [====>.........................] - ETA: 0s - loss: 0.3204 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 64/254 [======>.......................] - ETA: 0s - loss: 0.3218 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 80/254 [========>.....................] - ETA: 0s - loss: 0.3252 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:55 +0000\tmaster-replica-0\t\t 96/254 [==========>...................] - ETA: 0s - loss: 0.3242 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t112/254 [============>.................] - ETA: 0s - loss: 0.3230 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t129/254 [==============>...............] - ETA: 0s - loss: 0.3256 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t144/254 [================>.............] - ETA: 0s - loss: 0.3255 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t162/254 [==================>...........] - ETA: 0s - loss: 0.3251 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t177/254 [===================>..........] - ETA: 0s - loss: 0.3249 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t195/254 [======================>.......] - ETA: 0s - loss: 0.3263 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t211/254 [=======================>......] - ETA: 0s - loss: 0.3261 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t227/254 [=========================>....] - ETA: 0s - loss: 0.3257 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3262 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.3265 - acc: 0.8503 - val_loss: 0.3399 - val_acc: 0.8413\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\tEpoch 00015: LearningRateScheduler reducing learning rate to 0.0100006103515625.\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\tEpoch 15/20\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1s - loss: 0.2755 - acc: 0.859\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 14/254 [>.............................] - ETA: 0s - loss: 0.3182 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 27/254 [==>...........................] - ETA: 0s - loss: 0.3285 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 39/254 [===>..........................] - ETA: 0s - loss: 0.3268 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 47/254 [====>.........................] - ETA: 0s - loss: 0.3321 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 59/254 [=====>........................] - ETA: 0s - loss: 0.3231 - acc: 0.853\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 72/254 [=======>......................] - ETA: 0s - loss: 0.3262 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t 86/254 [=========>....................] - ETA: 0s - loss: 0.3273 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:56 +0000\tmaster-replica-0\t\t101/254 [==========>...................] - ETA: 0s - loss: 0.3294 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t117/254 [============>.................] - ETA: 0s - loss: 0.3276 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t131/254 [==============>...............] - ETA: 0s - loss: 0.3263 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t148/254 [================>.............] - ETA: 0s - loss: 0.3250 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t165/254 [==================>...........] - ETA: 0s - loss: 0.3260 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t182/254 [====================>.........] - ETA: 0s - loss: 0.3261 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t201/254 [======================>.......] - ETA: 0s - loss: 0.3259 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t218/254 [========================>.....] - ETA: 0s - loss: 0.3261 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t237/254 [==========================>...] - ETA: 0s - loss: 0.3266 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.3285 - acc: 0.8496 - val_loss: 0.3191 - val_acc: 0.8499\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\tEpoch 00016: LearningRateScheduler reducing learning rate to 0.01000030517578125.\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\tEpoch 16/20\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.3200 - acc: 0.828\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t 20/254 [=>............................] - ETA: 0s - loss: 0.3262 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t 40/254 [===>..........................] - ETA: 0s - loss: 0.3369 - acc: 0.843\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t 59/254 [=====>........................] - ETA: 0s - loss: 0.3348 - acc: 0.845\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t 79/254 [========>.....................] - ETA: 0s - loss: 0.3302 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t 98/254 [==========>...................] - ETA: 0s - loss: 0.3292 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t117/254 [============>.................] - ETA: 0s - loss: 0.3301 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t134/254 [==============>...............] - ETA: 0s - loss: 0.3275 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t151/254 [================>.............] - ETA: 0s - loss: 0.3253 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t167/254 [==================>...........] - ETA: 0s - loss: 0.3259 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:57 +0000\tmaster-replica-0\t\t185/254 [====================>.........] - ETA: 0s - loss: 0.3271 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t205/254 [=======================>......] - ETA: 0s - loss: 0.3292 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t224/254 [=========================>....] - ETA: 0s - loss: 0.3284 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t244/254 [===========================>..] - ETA: 0s - loss: 0.3271 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3278 - acc: 0.8495 - val_loss: 0.3245 - val_acc: 0.8537\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\tEpoch 00017: LearningRateScheduler reducing learning rate to 0.010000152587890625.\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\tEpoch 17/20\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.2615 - acc: 0.882\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t 19/254 [=>............................] - ETA: 0s - loss: 0.3367 - acc: 0.844\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t 37/254 [===>..........................] - ETA: 0s - loss: 0.3328 - acc: 0.846\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t 55/254 [=====>........................] - ETA: 0s - loss: 0.3300 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t 71/254 [=======>......................] - ETA: 0s - loss: 0.3255 - acc: 0.853\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t 88/254 [=========>....................] - ETA: 0s - loss: 0.3239 - acc: 0.853\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t105/254 [===========>..................] - ETA: 0s - loss: 0.3245 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t122/254 [=============>................] - ETA: 0s - loss: 0.3234 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t139/254 [===============>..............] - ETA: 0s - loss: 0.3218 - acc: 0.852\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t156/254 [=================>............] - ETA: 0s - loss: 0.3244 - acc: 0.851\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t172/254 [===================>..........] - ETA: 0s - loss: 0.3267 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t188/254 [=====================>........] - ETA: 0s - loss: 0.3286 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t203/254 [======================>.......] - ETA: 0s - loss: 0.3278 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:58 +0000\tmaster-replica-0\t\t219/254 [========================>.....] - ETA: 0s - loss: 0.3263 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t236/254 [==========================>...] - ETA: 0s - loss: 0.3270 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3270 - acc: 0.8500 - val_loss: 0.3272 - val_acc: 0.8521\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\tEpoch 00018: LearningRateScheduler reducing learning rate to 0.010000076293945313.\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\tEpoch 18/20\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1s - loss: 0.3422 - acc: 0.828\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 16/254 [>.............................] - ETA: 0s - loss: 0.3106 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 32/254 [==>...........................] - ETA: 0s - loss: 0.3128 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 47/254 [====>.........................] - ETA: 0s - loss: 0.3246 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 63/254 [======>.......................] - ETA: 0s - loss: 0.3241 - acc: 0.850\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 79/254 [========>.....................] - ETA: 0s - loss: 0.3233 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t 95/254 [==========>...................] - ETA: 0s - loss: 0.3282 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t111/254 [============>.................] - ETA: 0s - loss: 0.3282 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t126/254 [=============>................] - ETA: 0s - loss: 0.3266 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t145/254 [================>.............] - ETA: 0s - loss: 0.3273 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t163/254 [==================>...........] - ETA: 0s - loss: 0.3287 - acc: 0.847\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t178/254 [====================>.........] - ETA: 0s - loss: 0.3267 - acc: 0.848\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t194/254 [=====================>........] - ETA: 0s - loss: 0.3253 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t209/254 [=======================>......] - ETA: 0s - loss: 0.3248 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t224/254 [=========================>....] - ETA: 0s - loss: 0.3249 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t240/254 [===========================>..] - ETA: 0s - loss: 0.3254 - acc: 0.849\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.3244 - acc: 0.8493 - val_loss: 0.3271 - val_acc: 0.8508\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\tEpoch 00019: LearningRateScheduler reducing learning rate to 0.010000038146972657.\n",
      "INFO\t2019-03-27 17:56:59 +0000\tmaster-replica-0\t\tEpoch 19/20\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 1s - loss: 0.4041 - acc: 0.851\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 16/254 [>.............................] - ETA: 0s - loss: 0.3522 - acc: 0.846\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 33/254 [==>...........................] - ETA: 0s - loss: 0.3445 - acc: 0.849\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 50/254 [====>.........................] - ETA: 0s - loss: 0.3336 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 65/254 [======>.......................] - ETA: 0s - loss: 0.3331 - acc: 0.852\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 84/254 [========>.....................] - ETA: 0s - loss: 0.3301 - acc: 0.852\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t102/254 [===========>..................] - ETA: 0s - loss: 0.3289 - acc: 0.851\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t118/254 [============>.................] - ETA: 0s - loss: 0.3290 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t134/254 [==============>...............] - ETA: 0s - loss: 0.3310 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t154/254 [=================>............] - ETA: 0s - loss: 0.3283 - acc: 0.851\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t173/254 [===================>..........] - ETA: 0s - loss: 0.3270 - acc: 0.851\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t190/254 [=====================>........] - ETA: 0s - loss: 0.3266 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t201/254 [======================>.......] - ETA: 0s - loss: 0.3260 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t217/254 [========================>.....] - ETA: 0s - loss: 0.3281 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t231/254 [==========================>...] - ETA: 0s - loss: 0.3281 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t242/254 [===========================>..] - ETA: 0s - loss: 0.3270 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 4ms/step - loss: 0.3281 - acc: 0.8504 - val_loss: 0.3239 - val_acc: 0.8521\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\tEpoch 00020: LearningRateScheduler reducing learning rate to 0.010000019073486329.\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\tEpoch 20/20\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t  1/254 [..............................] - ETA: 0s - loss: 0.2766 - acc: 0.867\n",
      "INFO\t2019-03-27 17:57:00 +0000\tmaster-replica-0\t\t 20/254 [=>............................] - ETA: 0s - loss: 0.3260 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t 37/254 [===>..........................] - ETA: 0s - loss: 0.3271 - acc: 0.847\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t 58/254 [=====>........................] - ETA: 0s - loss: 0.3302 - acc: 0.847\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t 79/254 [========>.....................] - ETA: 0s - loss: 0.3354 - acc: 0.847\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t 98/254 [==========>...................] - ETA: 0s - loss: 0.3344 - acc: 0.846\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t119/254 [=============>................] - ETA: 0s - loss: 0.3318 - acc: 0.849\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t139/254 [===============>..............] - ETA: 0s - loss: 0.3314 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t156/254 [=================>............] - ETA: 0s - loss: 0.3316 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t175/254 [===================>..........] - ETA: 0s - loss: 0.3317 - acc: 0.851\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t192/254 [=====================>........] - ETA: 0s - loss: 0.3309 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t212/254 [========================>.....] - ETA: 0s - loss: 0.3296 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t231/254 [==========================>...] - ETA: 0s - loss: 0.3282 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t250/254 [============================>.] - ETA: 0s - loss: 0.3294 - acc: 0.850\n",
      "INFO\t2019-03-27 17:57:01 +0000\tmaster-replica-0\t\t254/254 [==============================] - 1s 3ms/step - loss: 0.3294 - acc: 0.8508 - val_loss: 0.3282 - val_acc: 0.8519\n",
      "WARNING\t2019-03-27 17:57:03 +0000\tmaster-replica-0\t\tThis model was compiled with a Keras optimizer (<tensorflow.python.keras.optimizers.RMSprop object at 0x7f3904fa0518>) but is being saved in TensorFlow format with `save_weights`. The model's weights will be saved, but unlike with TensorFlow optimizers in the TensorFlow format the optimizer's state will not be saved.\n",
      "WARNING\t2019-03-27 17:57:03 +0000\tmaster-replica-0\t\t\n",
      "WARNING\t2019-03-27 17:57:03 +0000\tmaster-replica-0\t\tConsider using a TensorFlow optimizer from `tf.train`.\n",
      "WARNING\t2019-03-27 17:57:05 +0000\tmaster-replica-0\t\tFrom /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/network.py:1436: update_checkpoint_state (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "WARNING\t2019-03-27 17:57:05 +0000\tmaster-replica-0\t\tInstructions for updating:\n",
      "WARNING\t2019-03-27 17:57:05 +0000\tmaster-replica-0\t\tUse tf.train.CheckpointManager to manage checkpoints rather than manually editing the Checkpoint proto.\n",
      "WARNING\t2019-03-27 17:57:05 +0000\tmaster-replica-0\t\tModel was compiled with an optimizer, but the optimizer is not from `tf.train` (e.g. `tf.train.AdagradOptimizer`). Only the serving graph was exported. The train and evaluate graphs were not added to the SavedModel.\n",
      "WARNING\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tFrom /usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "WARNING\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tInstructions for updating:\n",
      "WARNING\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tThis function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSignatures INCLUDED in export for Classify: None\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSignatures INCLUDED in export for Eval: None\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSignatures INCLUDED in export for Predict: ['serving_default']\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSignatures INCLUDED in export for Regress: None\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSignatures INCLUDED in export for Train: None\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tNo assets to save.\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tNo assets to write.\n",
      "INFO\t2019-03-27 17:57:10 +0000\tmaster-replica-0\t\tSavedModel written to: gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/saved_model.pb\n",
      "INFO\t2019-03-27 17:57:11 +0000\tmaster-replica-0\t\tModel exported to:  gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421\n",
      "INFO\t2019-03-27 17:57:11 +0000\tmaster-replica-0\t\tModule completed; cleaning up.\n",
      "INFO\t2019-03-27 17:57:11 +0000\tmaster-replica-0\t\tClean up finished.\n",
      "INFO\t2019-03-27 17:57:11 +0000\tmaster-replica-0\t\tTask completed successfully.\n",
      "endTime: '2019-03-27T18:01:46'\n",
      "jobId: my_first_keras_job\n",
      "startTime: '2019-03-27T17:55:34'\n",
      "state: SUCCEEDED\n"
     ]
    }
   ],
   "source": [
    "! gcloud ai-platform jobs submit training $JOB_NAME \\\n",
    "  --package-path trainer/ \\\n",
    "  --module-name trainer.task \\\n",
    "  --region $REGION \\\n",
    "  --python-version 3.7 \\\n",
    "  --runtime-version 1.15 \\\n",
    "  --job-dir $JOB_DIR \\\n",
    "  --stream-logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gAO6-zv6osJ8"
   },
   "source": [
    "## Part 2. Quickstart for online predictions in AI Platform\n",
    "\n",
    "This section shows how to use AI Platform and your trained model from Part 1\n",
    "to predict a person's income bracket from other Census information about them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Oi1xMGzLsjf_"
   },
   "source": [
    "### Create model and version resources in AI Platform\n",
    "\n",
    "To serve online predictions using the model you trained and exported in Part 1,\n",
    "create a *model* resource in AI Platform and a *version* resource\n",
    "within it. The version resource is what actually uses your trained model to\n",
    "serve predictions. This structure lets you adjust and retrain your model many times and\n",
    "organize all the versions together in AI Platform. Learn more about [models\n",
    "and\n",
    "versions](https://cloud.google.com/ai-platform/prediction/docs/projects-models-versions-jobs).\n",
    "\n",
    "While you specify `--region $REGION` in gcloud commands, you will use regional endpoint. You can also specify `--region global` to use global endpoint. Please note that you must create versions using the same endpoint as the one you use to create the model.  Learn more about available [regional endpoints](https://cloud.google.com/ai-platform/prediction/docs/regional-endpoints).\n",
    "\n",
    "First, name and create the model resource:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "SurMKEbBtc2W",
    "outputId": "28321881-0678-4805-a05c-290b8cfcad4e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created AI Platform model [projects/<your-project-id>/models/my_first_keras_model].\n"
     ]
    }
   ],
   "source": [
    "MODEL_NAME = \"my_first_keras_model\"\n",
    "\n",
    "! gcloud ai-platform models create $MODEL_NAME \\\n",
    "  --region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c5YLJugmt-Wm"
   },
   "source": [
    "Next, create the model version. The training job from Part 1 exported a timestamped\n",
    "[TensorFlow SavedModel\n",
    "directory](https://www.tensorflow.org/guide/saved_model#structure_of_a_savedmodel_directory)\n",
    "to your Cloud Storage bucket. AI Platform uses this directory to create a\n",
    "model version. Learn more about [SavedModel and\n",
    "AI Platform](https://cloud.google.com/ml-engine/docs/tensorflow/deploying-models).\n",
    "\n",
    "You may be able to find the path to this directory in your training job's logs.\n",
    "Look for a line like:\n",
    "\n",
    "```\n",
    "Model exported to:  gs://<your-bucket-name>/keras-job-dir/keras_export/1545439782\n",
    "```\n",
    "\n",
    "Execute the following command to identify your SavedModel directory and use it to create a model version resource:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "NYfK78654CVm"
   },
   "outputs": [],
   "source": [
    "MODEL_VERSION = \"v1\"\n",
    "\n",
    "# Get a list of directories in the `keras_export` parent directory\n",
    "KERAS_EXPORT_DIRS = ! gsutil ls $JOB_DIR/keras_export/\n",
    "\n",
    "# Update the directory as needed, in case you've trained\n",
    "# multiple times\n",
    "SAVED_MODEL_PATH = keras_export\n",
    "\n",
    "# Create model version based on that SavedModel directory\n",
    "! gcloud ai-platform versions create $MODEL_VERSION \\\n",
    "  --region $REGION \\\n",
    "  --model $MODEL_NAME \\\n",
    "  --runtime-version 1.15 \\\n",
    "  --python-version 3.7 \\\n",
    "  --framework tensorflow \\\n",
    "  --origin $SAVED_MODEL_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JzevJps9IOcU"
   },
   "source": [
    "### Prepare input for prediction\n",
    "\n",
    "To receive valid and useful predictions, you must preprocess input for prediction in the same way that training data was preprocessed. In a production\n",
    "system, you may want to create a preprocessing pipeline that can be used identically at training time and prediction time.\n",
    "\n",
    "For this exercise, use the training package's data-loading code to select a random sample from the evaluation data. This data is in the form that was used to evaluate accuracy after each epoch of training, so it can be used to send test predictions without further preprocessing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 669
    },
    "colab_type": "code",
    "id": "zuh7LWeWv_GT",
    "outputId": "c0e90fbe-ba9a-4095-943f-b8d17b4cf13f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1979</th>\n",
       "      <td>0.901213</td>\n",
       "      <td>1</td>\n",
       "      <td>1.525542</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.437544</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2430</th>\n",
       "      <td>-0.922154</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4214</th>\n",
       "      <td>-1.213893</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.030304</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>1.579979</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10389</th>\n",
       "      <td>-0.630415</td>\n",
       "      <td>3</td>\n",
       "      <td>0.358658</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.679647</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14525</th>\n",
       "      <td>-1.505632</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.586149</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15040</th>\n",
       "      <td>-0.119873</td>\n",
       "      <td>5</td>\n",
       "      <td>0.358658</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.841048</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8409</th>\n",
       "      <td>0.244801</td>\n",
       "      <td>3</td>\n",
       "      <td>1.525542</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>1.176475</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10628</th>\n",
       "      <td>0.098931</td>\n",
       "      <td>1</td>\n",
       "      <td>1.525542</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.886847</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10942</th>\n",
       "      <td>0.390670</td>\n",
       "      <td>5</td>\n",
       "      <td>-0.030304</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>4.727315</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5129</th>\n",
       "      <td>1.120017</td>\n",
       "      <td>3</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2096</th>\n",
       "      <td>-1.286827</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.030304</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-1.648058</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12463</th>\n",
       "      <td>-0.703350</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>4.502280</td>\n",
       "      <td>-0.437544</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8528</th>\n",
       "      <td>0.536539</td>\n",
       "      <td>3</td>\n",
       "      <td>1.525542</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7093</th>\n",
       "      <td>-1.359762</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12565</th>\n",
       "      <td>0.536539</td>\n",
       "      <td>3</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5655</th>\n",
       "      <td>1.338821</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2322</th>\n",
       "      <td>0.682409</td>\n",
       "      <td>3</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12652</th>\n",
       "      <td>0.025997</td>\n",
       "      <td>3</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>0.369465</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4755</th>\n",
       "      <td>-0.411611</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>1.176475</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4413</th>\n",
       "      <td>0.390670</td>\n",
       "      <td>6</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            age  workclass  education_num  marital_status  occupation  \\\n",
       "1979   0.901213          1       1.525542               2           9   \n",
       "2430  -0.922154          3      -0.419265               4           2   \n",
       "4214  -1.213893          3      -0.030304               4          10   \n",
       "10389 -0.630415          3       0.358658               4           0   \n",
       "14525 -1.505632          3      -1.586149               4           7   \n",
       "15040 -0.119873          5       0.358658               2           2   \n",
       "8409   0.244801          3       1.525542               2           9   \n",
       "10628  0.098931          1       1.525542               2           9   \n",
       "10942  0.390670          5      -0.030304               2           4   \n",
       "5129   1.120017          3       1.136580               2          12   \n",
       "2096  -1.286827          3      -0.030304               4          11   \n",
       "12463 -0.703350          3      -0.419265               2           7   \n",
       "8528   0.536539          3       1.525542               4           3   \n",
       "7093  -1.359762          3      -0.419265               4           6   \n",
       "12565  0.536539          3       1.136580               0          11   \n",
       "5655   1.338821          3      -0.419265               2           2   \n",
       "2322   0.682409          3       1.136580               0          12   \n",
       "12652  0.025997          3       1.136580               2          11   \n",
       "4755  -0.411611          3      -0.419265               2          11   \n",
       "4413   0.390670          6       1.136580               4           4   \n",
       "\n",
       "       relationship  race  capital_gain  capital_loss  hours_per_week  \\\n",
       "1979              0     4     -0.144792     -0.217132       -0.437544   \n",
       "2430              3     4     -0.144792     -0.217132       -0.034039   \n",
       "4214              1     4     -0.144792     -0.217132        1.579979   \n",
       "10389             3     4     -0.144792     -0.217132       -0.679647   \n",
       "14525             3     0     -0.144792     -0.217132       -0.034039   \n",
       "15040             0     4     -0.144792     -0.217132       -0.841048   \n",
       "8409              0     4     -0.144792     -0.217132        1.176475   \n",
       "10628             0     4      0.886847     -0.217132       -0.034039   \n",
       "10942             0     4     -0.144792     -0.217132        4.727315   \n",
       "5129              0     4     -0.144792     -0.217132       -0.034039   \n",
       "2096              3     4     -0.144792     -0.217132       -1.648058   \n",
       "12463             5     4     -0.144792      4.502280       -0.437544   \n",
       "8528              4     4     -0.144792     -0.217132       -0.034039   \n",
       "7093              3     2     -0.144792     -0.217132       -0.034039   \n",
       "12565             2     2     -0.144792     -0.217132       -0.034039   \n",
       "5655              0     4     -0.144792     -0.217132       -0.034039   \n",
       "2322              3     4     -0.144792     -0.217132       -0.034039   \n",
       "12652             0     4     -0.144792     -0.217132        0.369465   \n",
       "4755              0     4     -0.144792     -0.217132        1.176475   \n",
       "4413              1     4     -0.144792     -0.217132       -0.034039   \n",
       "\n",
       "       native_country  \n",
       "1979               38  \n",
       "2430               38  \n",
       "4214               38  \n",
       "10389              38  \n",
       "14525              38  \n",
       "15040              38  \n",
       "8409                6  \n",
       "10628              38  \n",
       "10942              38  \n",
       "5129               38  \n",
       "2096               38  \n",
       "12463              38  \n",
       "8528               38  \n",
       "7093               38  \n",
       "12565              38  \n",
       "5655               38  \n",
       "2322               38  \n",
       "12652              38  \n",
       "4755               38  \n",
       "4413               38  "
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trainer import util\n",
    "\n",
    "_, _, eval_x, eval_y = util.load_data()\n",
    "\n",
    "prediction_input = eval_x.sample(20)\n",
    "prediction_targets = eval_y[prediction_input.index]\n",
    "\n",
    "prediction_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HV1JhIWkn1t5"
   },
   "source": [
    "Notice that categorical fields, like `occupation`,  have already been converted to integers (with the same mapping that was used for training). Numerical fields, like `age`, have been scaled to a\n",
    "[z-score](https://developers.google.com/machine-learning/crash-course/representation/cleaning-data). Some fields have been dropped from the original\n",
    "data. Compare the prediction input with the raw data for the same examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 669
    },
    "colab_type": "code",
    "id": "fvRzpDgugqQr",
    "outputId": "4158ec9d-88d6-429e-f8cc-8582415b3b8d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>gender</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1979</th>\n",
       "      <td>51</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>99064</td>\n",
       "      <td>Masters</td>\n",
       "      <td>14</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2430</th>\n",
       "      <td>26</td>\n",
       "      <td>Private</td>\n",
       "      <td>197967</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4214</th>\n",
       "      <td>22</td>\n",
       "      <td>Private</td>\n",
       "      <td>221694</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10389</th>\n",
       "      <td>30</td>\n",
       "      <td>Private</td>\n",
       "      <td>96480</td>\n",
       "      <td>Assoc-voc</td>\n",
       "      <td>11</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14525</th>\n",
       "      <td>18</td>\n",
       "      <td>Private</td>\n",
       "      <td>146225</td>\n",
       "      <td>10th</td>\n",
       "      <td>6</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Amer-Indian-Eskimo</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15040</th>\n",
       "      <td>37</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>50096</td>\n",
       "      <td>Assoc-voc</td>\n",
       "      <td>11</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8409</th>\n",
       "      <td>42</td>\n",
       "      <td>Private</td>\n",
       "      <td>102988</td>\n",
       "      <td>Masters</td>\n",
       "      <td>14</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>Ecuador</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10628</th>\n",
       "      <td>40</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>284086</td>\n",
       "      <td>Masters</td>\n",
       "      <td>14</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10942</th>\n",
       "      <td>44</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>52505</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>99</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5129</th>\n",
       "      <td>54</td>\n",
       "      <td>Private</td>\n",
       "      <td>106728</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2096</th>\n",
       "      <td>21</td>\n",
       "      <td>Private</td>\n",
       "      <td>190916</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12463</th>\n",
       "      <td>29</td>\n",
       "      <td>Private</td>\n",
       "      <td>197565</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>1902</td>\n",
       "      <td>35</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8528</th>\n",
       "      <td>46</td>\n",
       "      <td>Private</td>\n",
       "      <td>193188</td>\n",
       "      <td>Masters</td>\n",
       "      <td>14</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7093</th>\n",
       "      <td>20</td>\n",
       "      <td>Private</td>\n",
       "      <td>273147</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12565</th>\n",
       "      <td>46</td>\n",
       "      <td>Private</td>\n",
       "      <td>203653</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Other-relative</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5655</th>\n",
       "      <td>57</td>\n",
       "      <td>Private</td>\n",
       "      <td>174662</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2322</th>\n",
       "      <td>48</td>\n",
       "      <td>Private</td>\n",
       "      <td>232149</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12652</th>\n",
       "      <td>39</td>\n",
       "      <td>Private</td>\n",
       "      <td>82521</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4755</th>\n",
       "      <td>33</td>\n",
       "      <td>Private</td>\n",
       "      <td>330715</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4413</th>\n",
       "      <td>44</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>128586</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age         workclass  fnlwgt     education  education_num  \\\n",
       "1979    51         Local-gov   99064       Masters             14   \n",
       "2430    26           Private  197967       HS-grad              9   \n",
       "4214    22           Private  221694  Some-college             10   \n",
       "10389   30           Private   96480     Assoc-voc             11   \n",
       "14525   18           Private  146225          10th              6   \n",
       "15040   37  Self-emp-not-inc   50096     Assoc-voc             11   \n",
       "8409    42           Private  102988       Masters             14   \n",
       "10628   40         Local-gov  284086       Masters             14   \n",
       "10942   44  Self-emp-not-inc   52505  Some-college             10   \n",
       "5129    54           Private  106728     Bachelors             13   \n",
       "2096    21           Private  190916  Some-college             10   \n",
       "12463   29           Private  197565       HS-grad              9   \n",
       "8528    46           Private  193188       Masters             14   \n",
       "7093    20           Private  273147       HS-grad              9   \n",
       "12565   46           Private  203653     Bachelors             13   \n",
       "5655    57           Private  174662       HS-grad              9   \n",
       "2322    48           Private  232149     Bachelors             13   \n",
       "12652   39           Private   82521     Bachelors             13   \n",
       "4755    33           Private  330715       HS-grad              9   \n",
       "4413    44         State-gov  128586     Bachelors             13   \n",
       "\n",
       "           marital_status         occupation    relationship  \\\n",
       "1979   Married-civ-spouse     Prof-specialty         Husband   \n",
       "2430        Never-married       Craft-repair       Own-child   \n",
       "4214        Never-married    Protective-serv   Not-in-family   \n",
       "10389       Never-married       Adm-clerical       Own-child   \n",
       "14525       Never-married      Other-service       Own-child   \n",
       "15040  Married-civ-spouse       Craft-repair         Husband   \n",
       "8409   Married-civ-spouse     Prof-specialty         Husband   \n",
       "10628  Married-civ-spouse     Prof-specialty         Husband   \n",
       "10942  Married-civ-spouse    Farming-fishing         Husband   \n",
       "5129   Married-civ-spouse       Tech-support         Husband   \n",
       "2096        Never-married              Sales       Own-child   \n",
       "12463  Married-civ-spouse      Other-service            Wife   \n",
       "8528        Never-married    Exec-managerial       Unmarried   \n",
       "7093        Never-married  Machine-op-inspct       Own-child   \n",
       "12565            Divorced              Sales  Other-relative   \n",
       "5655   Married-civ-spouse       Craft-repair         Husband   \n",
       "2322             Divorced       Tech-support       Own-child   \n",
       "12652  Married-civ-spouse              Sales         Husband   \n",
       "4755   Married-civ-spouse              Sales         Husband   \n",
       "4413        Never-married    Farming-fishing   Not-in-family   \n",
       "\n",
       "                     race  gender  capital_gain  capital_loss  hours_per_week  \\\n",
       "1979                White    Male             0             0              35   \n",
       "2430                White    Male             0             0              40   \n",
       "4214                White    Male             0             0              60   \n",
       "10389               White  Female             0             0              32   \n",
       "14525  Amer-Indian-Eskimo  Female             0             0              40   \n",
       "15040               White    Male             0             0              30   \n",
       "8409                White    Male             0             0              55   \n",
       "10628               White    Male          7688             0              40   \n",
       "10942               White    Male             0             0              99   \n",
       "5129                White    Male             0             0              40   \n",
       "2096                White  Female             0             0              20   \n",
       "12463               White  Female             0          1902              35   \n",
       "8528                White    Male             0             0              40   \n",
       "7093                Black    Male             0             0              40   \n",
       "12565               Black    Male             0             0              40   \n",
       "5655                White    Male             0             0              40   \n",
       "2322                White  Female             0             0              40   \n",
       "12652               White    Male             0             0              45   \n",
       "4755                White    Male             0             0              55   \n",
       "4413                White    Male             0             0              40   \n",
       "\n",
       "      native_country income_bracket  \n",
       "1979   United-States          <=50K  \n",
       "2430   United-States          <=50K  \n",
       "4214   United-States          <=50K  \n",
       "10389  United-States          <=50K  \n",
       "14525  United-States          <=50K  \n",
       "15040  United-States          <=50K  \n",
       "8409         Ecuador           >50K  \n",
       "10628  United-States           >50K  \n",
       "10942  United-States          <=50K  \n",
       "5129   United-States          <=50K  \n",
       "2096   United-States          <=50K  \n",
       "12463  United-States           >50K  \n",
       "8528   United-States          <=50K  \n",
       "7093   United-States          <=50K  \n",
       "12565  United-States          <=50K  \n",
       "5655   United-States          <=50K  \n",
       "2322   United-States          <=50K  \n",
       "12652  United-States           >50K  \n",
       "4755   United-States          <=50K  \n",
       "4413   United-States          <=50K  "
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "_, eval_file_path = util.download(util.DATA_DIR)\n",
    "raw_eval_data = pd.read_csv(eval_file_path,\n",
    "                            names=util._CSV_COLUMNS,\n",
    "                            na_values='?')\n",
    "\n",
    "raw_eval_data.iloc[prediction_input.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HFAbbH6ksG6s"
   },
   "source": [
    "Export the prediction input to a newline-delimited JSON file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "Yl3JALtnsGj-",
    "outputId": "f6d961a8-1ce8-4084-abb3-317b8d4336db"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.9012127751273994, 1.0, 1.525541514460902, 2.0, 9.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.43754385253479555, 38.0]\n",
      "[-0.9221541171760282, 3.0, -0.4192650914017433, 4.0, 2.0, 3.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[-1.2138928199445767, 3.0, -0.030303770229214273, 4.0, 10.0, 1.0, 4.0, -0.14479173735784842, -0.21713186390175285, 1.5799792247041626, 38.0]\n",
      "[-0.6304154144074798, 3.0, 0.35865755094331475, 4.0, 0.0, 3.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.6796466218034705, 38.0]\n",
      "[-1.5056315227131252, 3.0, -1.5861490549193304, 4.0, 7.0, 3.0, 0.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[-0.11987268456252011, 5.0, 0.35865755094331475, 2.0, 2.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.8410484679825871, 38.0]\n",
      "[0.24480069389816542, 3.0, 1.525541514460902, 2.0, 9.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, 1.176474609256371, 6.0]\n",
      "[0.0989313425138912, 1.0, 1.525541514460902, 2.0, 9.0, 0.0, 4.0, 0.8868473744801746, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[0.39067004528243965, 5.0, -0.030303770229214273, 2.0, 4.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, 4.7273152251969375, 38.0]\n",
      "[1.1200168022038106, 3.0, 1.1365801932883728, 2.0, 12.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[-1.2868274956367138, 3.0, -0.030303770229214273, 4.0, 11.0, 3.0, 4.0, -0.14479173735784842, -0.21713186390175285, -1.6480576988781703, 38.0]\n",
      "[-0.7033500900996169, 3.0, -0.4192650914017433, 2.0, 7.0, 5.0, 4.0, -0.14479173735784842, 4.5022796885373735, -0.43754385253479555, 38.0]\n",
      "[0.5365393966667138, 3.0, 1.525541514460902, 4.0, 3.0, 4.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[-1.3597621713288508, 3.0, -0.4192650914017433, 4.0, 6.0, 3.0, 2.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[0.5365393966667138, 3.0, 1.1365801932883728, 0.0, 11.0, 2.0, 2.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[1.338820829280222, 3.0, -0.4192650914017433, 2.0, 2.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[0.6824087480509881, 3.0, 1.1365801932883728, 0.0, 12.0, 3.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n",
      "[0.0259966668217541, 3.0, 1.1365801932883728, 2.0, 11.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, 0.3694653783607877, 38.0]\n",
      "[-0.4116113873310685, 3.0, -0.4192650914017433, 2.0, 11.0, 0.0, 4.0, -0.14479173735784842, -0.21713186390175285, 1.176474609256371, 38.0]\n",
      "[0.39067004528243965, 6.0, 1.1365801932883728, 4.0, 4.0, 1.0, 4.0, -0.14479173735784842, -0.21713186390175285, -0.03403923708700391, 38.0]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "with open('prediction_input.json', 'w') as json_file:\n",
    "  for row in prediction_input.values.tolist():\n",
    "    json.dump(row, json_file)\n",
    "    json_file.write('\\n')\n",
    "\n",
    "! cat prediction_input.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kVvEmazFKUCp"
   },
   "source": [
    "The `gcloud` command-line tool accepts newline-delimited JSON for online\n",
    "prediction, and this particular Keras model expects a flat list of\n",
    "numbers for each input example.\n",
    "\n",
    "AI Platform requires a different format when you make online prediction requests to the REST API without using the `gcloud` tool. The way you structure\n",
    "your model may also change how you must format data for prediction. Learn more\n",
    "about [formatting data for online\n",
    "prediction](https://cloud.google.com/ml-engine/docs/tensorflow/prediction-overview#prediction_input_data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "91xQnLqRN8A8"
   },
   "source": [
    "### Submit the online prediction request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KrDIoVLaG7zZ"
   },
   "source": [
    "Use `gcloud` to submit your online prediction request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "5gIcXDFwOERG",
    "outputId": "104e950d-5377-4ee1-f812-789ab4c09b90"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DENSE_4\n",
      "[0.6854287385940552]\n",
      "[0.011786997318267822]\n",
      "[0.037236183881759644]\n",
      "[0.016223609447479248]\n",
      "[0.0012015104293823242]\n",
      "[0.23621389269828796]\n",
      "[0.6174039244651794]\n",
      "[0.9822691679000854]\n",
      "[0.3815768361091614]\n",
      "[0.6715215444564819]\n",
      "[0.001094043254852295]\n",
      "[0.43077391386032104]\n",
      "[0.22132840752601624]\n",
      "[0.004075437784194946]\n",
      "[0.22736871242523193]\n",
      "[0.4111979305744171]\n",
      "[0.27328649163246155]\n",
      "[0.6981356143951416]\n",
      "[0.3309604525566101]\n",
      "[0.20807647705078125]\n"
     ]
    }
   ],
   "source": [
    "! gcloud ai-platform predict \\\n",
    "  --region $REGION \\\n",
    "  --model $MODEL_NAME \\\n",
    "  --version $MODEL_VERSION \\\n",
    "  --json-instances prediction_input.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n6U2DYKJGwdf"
   },
   "source": [
    "Since the model's last layer uses a [sigmoid function](https://developers.google.com/machine-learning/glossary/#sigmoid_function) for its activation, outputs between 0 and 0.5 represent negative predictions (\"<=50K\") and outputs between 0.5 and 1 represent positive ones (\">50K\").\n",
    "\n",
    "Do the predicted income brackets match the actual ones? Run the following cell\n",
    "to see the true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "w2Piq3PcGhaX",
    "outputId": "8fcc69d4-485b-4733-e594-0a92afa2c323"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.]], dtype=float32)"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction_targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GKP7F6-EDb5_"
   },
   "source": [
    "## Part 3. Developing the Keras model from scratch\n",
    "\n",
    "At this point, you have trained a machine learning model on AI Platform, deployed the trained model as a version resource on AI Platform, and received online predictions from the deployment. The next section walks through recreating the Keras code used to train your model. It covers the following parts of developing a machine learning model for use with AI Platform:\n",
    "\n",
    "* Downloading and preprocessing data\n",
    "* Designing and training the model\n",
    "* Visualizing training and exporting the trained model\n",
    "\n",
    "While this section provides more detailed insight to the tasks completed in previous parts, to learn more about using `tf.keras`, read [TensorFlow's guide to Keras](https://www.tensorflow.org/tutorials/keras). To learn more about structuring code as a training packge for AI Platform, read [Packaging a training application](https://cloud.google.com/ml-engine/docs/tensorflow/packaging-trainer) and reference the [complete training code](https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census/tf-keras), which is structured as a Python package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-VtUN0L5x4ql"
   },
   "source": [
    "### Import libraries and define constants\n",
    "\n",
    "First, import Python libraries required for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "RcxfR3GfscsA",
    "outputId": "11ea7f86-9030-4120-872b-0e232ec1c0b5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.6.7 (default, Oct 22 2018, 11:32:17) \n",
      "[GCC 8.2.0]\n",
      "1.15.2\n",
      "2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from six.moves import urllib\n",
    "import tempfile\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "# Examine software versions\n",
    "print(__import__('sys').version)\n",
    "print(tf.__version__)\n",
    "print(tf.keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xWZQbZQmx26U"
   },
   "source": [
    "Then, define some useful constants:\n",
    "\n",
    "* Information for downloading training and evaluation data\n",
    "* Information required for Pandas to interpret the data and convert categorical fields into numeric features\n",
    "* Hyperparameters for training, such as learning rate and batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cx4OXeXEsh_v"
   },
   "outputs": [],
   "source": [
    "### For downloading data ###\n",
    "\n",
    "# Storage directory\n",
    "DATA_DIR = os.path.join(tempfile.gettempdir(), 'census_data')\n",
    "\n",
    "# Download options.\n",
    "DATA_URL = 'https://storage.googleapis.com/cloud-samples-data/ai-platform' \\\n",
    "           '/census/data'\n",
    "TRAINING_FILE = 'adult.data.csv'\n",
    "EVAL_FILE = 'adult.test.csv'\n",
    "TRAINING_URL = '%s/%s' % (DATA_URL, TRAINING_FILE)\n",
    "EVAL_URL = '%s/%s' % (DATA_URL, EVAL_FILE)\n",
    "\n",
    "### For interpreting data ###\n",
    "\n",
    "# These are the features in the dataset.\n",
    "# Dataset information: https://archive.ics.uci.edu/ml/datasets/census+income\n",
    "_CSV_COLUMNS = [\n",
    "    'age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
    "    'marital_status', 'occupation', 'relationship', 'race', 'gender',\n",
    "    'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',\n",
    "    'income_bracket'\n",
    "]\n",
    "\n",
    "_CATEGORICAL_TYPES = {\n",
    "    'workclass': pd.api.types.CategoricalDtype(categories=[\n",
    "        'Federal-gov', 'Local-gov', 'Never-worked', 'Private', 'Self-emp-inc',\n",
    "        'Self-emp-not-inc', 'State-gov', 'Without-pay'\n",
    "    ]),\n",
    "    'marital_status': pd.api.types.CategoricalDtype(categories=[\n",
    "        'Divorced', 'Married-AF-spouse', 'Married-civ-spouse',\n",
    "        'Married-spouse-absent', 'Never-married', 'Separated', 'Widowed'\n",
    "    ]),\n",
    "    'occupation': pd.api.types.CategoricalDtype([\n",
    "        'Adm-clerical', 'Armed-Forces', 'Craft-repair', 'Exec-managerial',\n",
    "        'Farming-fishing', 'Handlers-cleaners', 'Machine-op-inspct',\n",
    "        'Other-service', 'Priv-house-serv', 'Prof-specialty', 'Protective-serv',\n",
    "        'Sales', 'Tech-support', 'Transport-moving'\n",
    "    ]),\n",
    "    'relationship': pd.api.types.CategoricalDtype(categories=[\n",
    "        'Husband', 'Not-in-family', 'Other-relative', 'Own-child', 'Unmarried',\n",
    "        'Wife'\n",
    "    ]),\n",
    "    'race': pd.api.types.CategoricalDtype(categories=[\n",
    "        'Amer-Indian-Eskimo', 'Asian-Pac-Islander', 'Black', 'Other', 'White'\n",
    "    ]),\n",
    "    'native_country': pd.api.types.CategoricalDtype(categories=[\n",
    "        'Cambodia', 'Canada', 'China', 'Columbia', 'Cuba', 'Dominican-Republic',\n",
    "        'Ecuador', 'El-Salvador', 'England', 'France', 'Germany', 'Greece',\n",
    "        'Guatemala', 'Haiti', 'Holand-Netherlands', 'Honduras', 'Hong', 'Hungary',\n",
    "        'India', 'Iran', 'Ireland', 'Italy', 'Jamaica', 'Japan', 'Laos', 'Mexico',\n",
    "        'Nicaragua', 'Outlying-US(Guam-USVI-etc)', 'Peru', 'Philippines', 'Poland',\n",
    "        'Portugal', 'Puerto-Rico', 'Scotland', 'South', 'Taiwan', 'Thailand',\n",
    "        'Trinadad&Tobago', 'United-States', 'Vietnam', 'Yugoslavia'\n",
    "    ]),\n",
    "    'income_bracket': pd.api.types.CategoricalDtype(categories=[\n",
    "        '<=50K', '>50K'\n",
    "    ])\n",
    "}\n",
    "\n",
    "# This is the label (target) we want to predict.\n",
    "_LABEL_COLUMN = 'income_bracket'\n",
    "\n",
    "### Hyperparameters for training ###\n",
    "\n",
    "# This the training batch size\n",
    "BATCH_SIZE = 128\n",
    "\n",
    "# This is the number of epochs (passes over the full training data)\n",
    "NUM_EPOCHS = 20\n",
    "\n",
    "# Define learning rate.\n",
    "LEARNING_RATE = .01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bSJjhQ8ZyDae"
   },
   "source": [
    "### Download and preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uafupKCgxazq"
   },
   "source": [
    "#### Download the data\n",
    "\n",
    "Next, define functions to download training and evaluation data. These functions also fix minor irregularities in the data's formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "iGorBTXWUZPy"
   },
   "outputs": [],
   "source": [
    "def _download_and_clean_file(filename, url):\n",
    "    \"\"\"Downloads data from url, and makes changes to match the CSV format.\n",
    "  \n",
    "    The CSVs may use spaces after the comma delimters (non-standard) or include\n",
    "    rows which do not represent well-formed examples. This function strips out\n",
    "    some of these problems.\n",
    "  \n",
    "    Args:\n",
    "      filename: filename to save url to\n",
    "      url: URL of resource to download\n",
    "    \"\"\"\n",
    "    temp_file, _ = urllib.request.urlretrieve(url)\n",
    "    with tf.io.gfile.GFile(temp_file, 'r') as temp_file_object:\n",
    "        with tf.io.gfile.GFile(filename, 'w') as file_object:\n",
    "            for line in temp_file_object:\n",
    "                line = line.strip()\n",
    "                line = line.replace(', ', ',')\n",
    "                if not line or ',' not in line:\n",
    "                    continue\n",
    "                if line[-1] == '.':\n",
    "                    line = line[:-1]\n",
    "                line += '\\n'\n",
    "                file_object.write(line)\n",
    "    tf.io.gfile.remove(temp_file)\n",
    "\n",
    "\n",
    "def download(data_dir):\n",
    "    \"\"\"Downloads census data if it is not already present.\n",
    "  \n",
    "    Args:\n",
    "      data_dir: directory where we will access/save the census data\n",
    "    \"\"\"\n",
    "    tf.io.gfile.makedirs(data_dir)\n",
    "\n",
    "    training_file_path = os.path.join(data_dir, TRAINING_FILE)\n",
    "    if not tf.io.gfile.exists(training_file_path):\n",
    "        _download_and_clean_file(training_file_path, TRAINING_URL)\n",
    "\n",
    "    eval_file_path = os.path.join(data_dir, EVAL_FILE)\n",
    "    if not tf.io.gfile.exists(eval_file_path):\n",
    "        _download_and_clean_file(eval_file_path, EVAL_URL)\n",
    "\n",
    "    return training_file_path, eval_file_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fuX5SyAOgYsG"
   },
   "source": [
    "Use those functions to download the data for training and verify that you have CSV files for training and evaluation: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "9Wii7NAss92J",
    "outputId": "dab93c34-27af-4dcb-91cc-e3ba9a8e9bd0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 5156\n",
      "-rw-r--r-- 1 root root 3518450 Mar 27 17:52 adult.data.csv\n",
      "-rw-r--r-- 1 root root 1758573 Mar 27 17:52 adult.test.csv\n"
     ]
    }
   ],
   "source": [
    "training_file_path, eval_file_path = download(DATA_DIR)\n",
    "\n",
    "# You should see 2 files: adult.data.csv and adult.test.csv\n",
    "!ls -l $DATA_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tHSJTrDygqpS"
   },
   "source": [
    "Next, load these files using Pandas and examine the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "KHWiAYCew28Q",
    "outputId": "36eb1311-815e-4800-9c6b-fc5ea88c39af"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>gender</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         workclass  fnlwgt  education  education_num  \\\n",
       "0   39         State-gov   77516  Bachelors             13   \n",
       "1   50  Self-emp-not-inc   83311  Bachelors             13   \n",
       "2   38           Private  215646    HS-grad              9   \n",
       "3   53           Private  234721       11th              7   \n",
       "4   28           Private  338409  Bachelors             13   \n",
       "\n",
       "       marital_status         occupation   relationship   race  gender  \\\n",
       "0       Never-married       Adm-clerical  Not-in-family  White    Male   \n",
       "1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n",
       "2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n",
       "3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n",
       "4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n",
       "\n",
       "   capital_gain  capital_loss  hours_per_week native_country income_bracket  \n",
       "0          2174             0              40  United-States          <=50K  \n",
       "1             0             0              13  United-States          <=50K  \n",
       "2             0             0              40  United-States          <=50K  \n",
       "3             0             0              40  United-States          <=50K  \n",
       "4             0             0              40           Cuba          <=50K  "
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This census data uses the value '?' for fields (column) that are missing data. \n",
    "# We use na_values to find ? and set it to NaN values.\n",
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\n",
    "\n",
    "train_df = pd.read_csv(training_file_path, names=_CSV_COLUMNS, na_values='?')\n",
    "eval_df = pd.read_csv(eval_file_path, names=_CSV_COLUMNS, na_values='?')\n",
    "\n",
    "# Here's what the data looks like before we preprocess the data.\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hhsa1-6qVD0n"
   },
   "source": [
    "#### Preprocess the data\n",
    "\n",
    "The first preprocessing step removes certain features from the data and\n",
    "converts categorical features to numerical values for use with Keras.\n",
    "\n",
    "Learn more about [feature engineering](https://developers.google.com/machine-learning/crash-course/representation/feature-engineering) and [bias in data](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WHbJecf1Bb2r"
   },
   "outputs": [],
   "source": [
    "UNUSED_COLUMNS = ['fnlwgt', 'education', 'gender']\n",
    "\n",
    "\n",
    "def preprocess(dataframe):\n",
    "    \"\"\"Converts categorical features to numeric. Removes unused columns.\n",
    "  \n",
    "    Args:\n",
    "      dataframe: Pandas dataframe with raw data\n",
    "  \n",
    "    Returns:\n",
    "      Dataframe with preprocessed data\n",
    "    \"\"\"\n",
    "    dataframe = dataframe.drop(columns=UNUSED_COLUMNS)\n",
    "\n",
    "    # Convert integer valued (numeric) columns to floating point\n",
    "    numeric_columns = dataframe.select_dtypes(['int64']).columns\n",
    "    dataframe[numeric_columns] = dataframe[numeric_columns].astype('float32')\n",
    "\n",
    "    # Convert categorical columns to numeric\n",
    "    cat_columns = dataframe.select_dtypes(['object']).columns\n",
    "    dataframe[cat_columns] = dataframe[cat_columns].apply(lambda x: x.astype(\n",
    "        _CATEGORICAL_TYPES[x.name]))\n",
    "    dataframe[cat_columns] = dataframe[cat_columns].apply(lambda x: x.cat.codes)\n",
    "    return dataframe\n",
    "\n",
    "\n",
    "prepped_train_df = preprocess(train_df)\n",
    "prepped_eval_df = preprocess(eval_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ce0Ln1P-mpwx"
   },
   "source": [
    "Run the following cell to see how preprocessing changed the data. Notice in particular that `income_bracket`, the label that you're training the model to predict, has changed from `<=50K` and `>50K` to `0` and `1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "YbMskdWmTCED",
    "outputId": "add41517-a4b6-41e2-9168-15d0f587c971"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39.0</td>\n",
       "      <td>6</td>\n",
       "      <td>13.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2174.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50.0</td>\n",
       "      <td>5</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38.0</td>\n",
       "      <td>3</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28.0</td>\n",
       "      <td>3</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  workclass  education_num  marital_status  occupation  relationship  \\\n",
       "0  39.0          6           13.0               4           0             1   \n",
       "1  50.0          5           13.0               2           3             0   \n",
       "2  38.0          3            9.0               0           5             1   \n",
       "3  53.0          3            7.0               2           5             0   \n",
       "4  28.0          3           13.0               2           9             5   \n",
       "\n",
       "   race  capital_gain  capital_loss  hours_per_week  native_country  \\\n",
       "0     4        2174.0           0.0            40.0              38   \n",
       "1     4           0.0           0.0            13.0              38   \n",
       "2     4           0.0           0.0            40.0              38   \n",
       "3     2           0.0           0.0            40.0              38   \n",
       "4     2           0.0           0.0            40.0               4   \n",
       "\n",
       "   income_bracket  \n",
       "0               0  \n",
       "1               0  \n",
       "2               0  \n",
       "3               0  \n",
       "4               0  "
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prepped_train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OCzBX6LuCmTT"
   },
   "source": [
    "Next, separate the data into features (\"x\") and labels (\"y\"), and reshape the label arrays into a format for use with `tf.data.Dataset` later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gPq7WY51GW6M"
   },
   "outputs": [],
   "source": [
    "# Split train and test data with labels.\n",
    "# The pop() method will extract (copy) and remove the label column from the dataframe\n",
    "train_x, train_y = prepped_train_df, prepped_train_df.pop(_LABEL_COLUMN)\n",
    "eval_x, eval_y = prepped_eval_df, prepped_eval_df.pop(_LABEL_COLUMN)\n",
    "\n",
    "# Reshape label columns for use with tf.data.Dataset\n",
    "train_y = np.asarray(train_y).astype('float32').reshape((-1, 1))\n",
    "eval_y = np.asarray(eval_y).astype('float32').reshape((-1, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A1tDH7_RDAIK"
   },
   "source": [
    "Scaling training data so each numerical feature column has a mean of 0 and a standard deviation of 1 [can improve your model](https://developers.google.com/machine-learning/crash-course/representation/cleaning-data).\n",
    "\n",
    "In a production system, you may want to save the means and standard deviations from your training set and use them to perform an identical transformation on test data at prediction time. For convenience in this exercise, temporarily combine the training and evaluation data to scale all of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OXAMwr3dCsqd"
   },
   "outputs": [],
   "source": [
    "def standardize(dataframe):\n",
    "    \"\"\"Scales numerical columns using their means and standard deviation to get\n",
    "    z-scores: the mean of each numerical column becomes 0, and the standard\n",
    "    deviation becomes 1. This can help the model converge during training.\n",
    "  \n",
    "    Args:\n",
    "      dataframe: Pandas dataframe\n",
    "  \n",
    "    Returns:\n",
    "      Input dataframe with the numerical columns scaled to z-scores\n",
    "    \"\"\"\n",
    "    dtypes = list(zip(dataframe.dtypes.index, map(str, dataframe.dtypes)))\n",
    "    # Normalize numeric columns.\n",
    "    for column, dtype in dtypes:\n",
    "        if dtype == 'float32':\n",
    "            dataframe[column] -= dataframe[column].mean()\n",
    "            dataframe[column] /= dataframe[column].std()\n",
    "    return dataframe\n",
    "\n",
    "\n",
    "# Join train_x and eval_x to normalize on overall means and standard\n",
    "# deviations. Then separate them again.\n",
    "all_x = pd.concat([train_x, eval_x], keys=['train', 'eval'])\n",
    "all_x = standardize(all_x)\n",
    "train_x, eval_x = all_x.xs('train'), all_x.xs('eval')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aqKZk6NawDe-"
   },
   "source": [
    "Finally, examine some of your fully preprocessed training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "6NZqE3Wlg1yg",
    "outputId": "6f782613-c20c-47e1-cbce-ac73ccdc1236"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.025997</td>\n",
       "      <td>6</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.146933</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.828278</td>\n",
       "      <td>5</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-2.212964</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.046938</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.419265</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.047082</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.197188</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.776285</td>\n",
       "      <td>3</td>\n",
       "      <td>1.136580</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.144792</td>\n",
       "      <td>-0.217132</td>\n",
       "      <td>-0.034039</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        age  workclass  education_num  marital_status  occupation  \\\n",
       "0  0.025997          6       1.136580               4           0   \n",
       "1  0.828278          5       1.136580               2           3   \n",
       "2 -0.046938          3      -0.419265               0           5   \n",
       "3  1.047082          3      -1.197188               2           5   \n",
       "4 -0.776285          3       1.136580               2           9   \n",
       "\n",
       "   relationship  race  capital_gain  capital_loss  hours_per_week  \\\n",
       "0             1     4      0.146933     -0.217132       -0.034039   \n",
       "1             0     4     -0.144792     -0.217132       -2.212964   \n",
       "2             1     4     -0.144792     -0.217132       -0.034039   \n",
       "3             0     2     -0.144792     -0.217132       -0.034039   \n",
       "4             5     2     -0.144792     -0.217132       -0.034039   \n",
       "\n",
       "   native_country  \n",
       "0              38  \n",
       "1              38  \n",
       "2              38  \n",
       "3              38  \n",
       "4               4  "
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify dataset features\n",
    "# Note how only the numeric fields (not categorical) have been standardized\n",
    "train_x.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V1bLJFrNxHXV"
   },
   "source": [
    "### Design and train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iWkmN8ZmwSXU"
   },
   "source": [
    "#### Create training and validation datasets\n",
    "\n",
    "Create an input function to convert features and labels into a\n",
    "[`tf.data.Dataset`](https://www.tensorflow.org/guide/datasets) for training or evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIfE-YaXwRaU"
   },
   "outputs": [],
   "source": [
    "def input_fn(features, labels, shuffle, num_epochs, batch_size):\n",
    "    \"\"\"Generates an input function to be used for model training.\n",
    "  \n",
    "    Args:\n",
    "      features: numpy array of features used for training or inference\n",
    "      labels: numpy array of labels for each example\n",
    "      shuffle: boolean for whether to shuffle the data or not (set True for\n",
    "        training, False for evaluation)\n",
    "      num_epochs: number of epochs to provide the data for\n",
    "      batch_size: batch size for training\n",
    "  \n",
    "    Returns:\n",
    "      A tf.data.Dataset that can provide data to the Keras model for training or\n",
    "        evaluation\n",
    "    \"\"\"\n",
    "    if labels is None:\n",
    "        inputs = features\n",
    "    else:\n",
    "        inputs = (features, labels)\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(inputs)\n",
    "\n",
    "    if shuffle:\n",
    "        dataset = dataset.shuffle(buffer_size=len(features))\n",
    "\n",
    "    # We call repeat after shuffling, rather than before, to prevent separate\n",
    "    # epochs from blending together.\n",
    "    dataset = dataset.repeat(num_epochs)\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f8paLJ_rF84R"
   },
   "source": [
    "Next, create these training and evaluation datasets.Use the `NUM_EPOCHS`\n",
    "and `BATCH_SIZE` hyperparameters defined previously to define how the training\n",
    "dataset provides examples to the model during training. Set up the validation\n",
    "dataset to provide all its examples in one batch, for a single validation step\n",
    "at the end of each training epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_xdC2P5EF7rH"
   },
   "outputs": [],
   "source": [
    "# Pass a numpy array by using DataFrame.values\n",
    "training_dataset = input_fn(features=train_x.values, \n",
    "                    labels=train_y, \n",
    "                    shuffle=True, \n",
    "                    num_epochs=NUM_EPOCHS, \n",
    "                    batch_size=BATCH_SIZE)\n",
    "\n",
    "num_eval_examples = eval_x.shape[0]\n",
    "\n",
    "# Pass a numpy array by using DataFrame.values\n",
    "validation_dataset = input_fn(features=eval_x.values, \n",
    "                    labels=eval_y, \n",
    "                    shuffle=False, \n",
    "                    num_epochs=NUM_EPOCHS, \n",
    "                    batch_size=num_eval_examples)                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vZrDBXnxggOH"
   },
   "source": [
    "#### Design a Keras Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JY_A0yPyqU08"
   },
   "source": [
    "Design your neural network using the [Keras Sequential API](https://www.tensorflow.org/guide/keras#sequential_model).\n",
    "\n",
    "This deep neural network (DNN) has several hidden layers, and the last layer uses a sigmoid activation function to output a value between 0 and 1:\n",
    "\n",
    "* The input layer has 100 units using the ReLU activation function.\n",
    "* The hidden layer has 75 units using the ReLU activation function.\n",
    "* The hidden layer has 50 units using the ReLU activation function.\n",
    "* The hidden layer has 25 units using the ReLU activation function.\n",
    "* The output layer has 1 units using a sigmoid activation function.\n",
    "* The optimizer uses the binary cross-entropy loss function, which is appropriate for a binary classification problem like this one.\n",
    "\n",
    "Feel free to change these layers to try to improve the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j_lsOhQAvzS-"
   },
   "outputs": [],
   "source": [
    "def create_keras_model(input_dim, learning_rate):\n",
    "    \"\"\"Creates Keras Model for Binary Classification.\n",
    "  \n",
    "    Args:\n",
    "      input_dim: How many features the input has\n",
    "      learning_rate: Learning rate for training\n",
    "  \n",
    "    Returns:\n",
    "      The compiled Keras model (still needs to be trained)\n",
    "    \"\"\"\n",
    "    Dense = tf.keras.layers.Dense\n",
    "    model = tf.keras.Sequential(\n",
    "      [\n",
    "          Dense(100, activation=tf.nn.relu, kernel_initializer='uniform',\n",
    "                  input_shape=(input_dim,)),\n",
    "          Dense(75, activation=tf.nn.relu),\n",
    "          Dense(50, activation=tf.nn.relu),\n",
    "          Dense(25, activation=tf.nn.relu),\n",
    "          Dense(1, activation=tf.nn.sigmoid)\n",
    "      ])\n",
    "    # Custom Optimizer:\n",
    "    # https://www.tensorflow.org/api_docs/python/tf/train/RMSPropOptimizer\n",
    "    optimizer = tf.keras.optimizers.RMSprop(\n",
    "        lr=learning_rate)\n",
    "\n",
    "    # Compile Keras model\n",
    "    model.compile(\n",
    "        loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mo5zTRNvBWv4"
   },
   "source": [
    "Next, create the Keras model object and examine its structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 391
    },
    "colab_type": "code",
    "id": "zdyOuwQcSjaY",
    "outputId": "4364f1a7-8c6a-492f-fc70-17e93a0a5a01"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 11\n",
      "Number of examples: 32561\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 100)               1200      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 75)                7575      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 50)                3800      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 26        \n",
      "=================================================================\n",
      "Total params: 13,876\n",
      "Trainable params: 13,876\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "num_train_examples, input_dim = train_x.shape\n",
    "print('Number of features: {}'.format(input_dim))\n",
    "print('Number of examples: {}'.format(num_train_examples))\n",
    "\n",
    "keras_model = create_keras_model(\n",
    "    input_dim=input_dim,\n",
    "    learning_rate=LEARNING_RATE)\n",
    "\n",
    "# Take a detailed look inside the model\n",
    "keras_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4cqeQttaHbZt"
   },
   "source": [
    "#### Train and evaluate the model\n",
    "\n",
    "Define a learning rate decay to encourage model paramaters to make smaller\n",
    "changes as training goes on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7eLa_Yoj2rhv"
   },
   "outputs": [],
   "source": [
    "# Setup Learning Rate decay.\n",
    "lr_decay_cb = tf.keras.callbacks.LearningRateScheduler(\n",
    "    lambda epoch: LEARNING_RATE + 0.02 * (0.5 ** (1 + epoch)),\n",
    "    verbose=True)\n",
    "\n",
    "# Setup TensorBoard callback.\n",
    "tensorboard_cb = tf.keras.callbacks.TensorBoard(\n",
    "      os.path.join(JOB_DIR, 'keras_tensorboard'),\n",
    "      histogram_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1fExnH6bhOSC"
   },
   "source": [
    "Finally, train the model. Provide the appropriate `steps_per_epoch` for the\n",
    "model to train on the entire training dataset (with `BATCH_SIZE` examples per step) during each epoch. And instruct the model to calculate validation\n",
    "accuracy with one big validation batch at the end of each epoch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1428
    },
    "colab_type": "code",
    "id": "MG4EvLiorMmZ",
    "outputId": "ca1a5e4a-026f-482b-9cf5-0a84819470c9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "\n",
      "Epoch 00001: LearningRateScheduler reducing learning rate to 0.02.\n",
      "Epoch 1/20\n",
      "254/254 [==============================] - 1s 5ms/step - loss: 0.6986 - acc: 0.7893 - val_loss: 0.3894 - val_acc: 0.8329\n",
      "\n",
      "Epoch 00002: LearningRateScheduler reducing learning rate to 0.015.\n",
      "Epoch 2/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3574 - acc: 0.8335 - val_loss: 0.3861 - val_acc: 0.8131\n",
      "\n",
      "Epoch 00003: LearningRateScheduler reducing learning rate to 0.0125.\n",
      "Epoch 3/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3422 - acc: 0.8404 - val_loss: 0.3304 - val_acc: 0.8445\n",
      "\n",
      "Epoch 00004: LearningRateScheduler reducing learning rate to 0.01125.\n",
      "Epoch 4/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3353 - acc: 0.8465 - val_loss: 0.3610 - val_acc: 0.8435\n",
      "\n",
      "Epoch 00005: LearningRateScheduler reducing learning rate to 0.010625.\n",
      "Epoch 5/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3315 - acc: 0.8457 - val_loss: 0.3288 - val_acc: 0.8445\n",
      "\n",
      "Epoch 00006: LearningRateScheduler reducing learning rate to 0.0103125.\n",
      "Epoch 6/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3294 - acc: 0.8475 - val_loss: 0.3331 - val_acc: 0.8489\n",
      "\n",
      "Epoch 00007: LearningRateScheduler reducing learning rate to 0.01015625.\n",
      "Epoch 7/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3296 - acc: 0.8476 - val_loss: 0.3296 - val_acc: 0.8508\n",
      "\n",
      "Epoch 00008: LearningRateScheduler reducing learning rate to 0.010078125.\n",
      "Epoch 8/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3287 - acc: 0.8486 - val_loss: 0.3254 - val_acc: 0.8494\n",
      "\n",
      "Epoch 00009: LearningRateScheduler reducing learning rate to 0.0100390625.\n",
      "Epoch 9/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3274 - acc: 0.8483 - val_loss: 0.3205 - val_acc: 0.8511\n",
      "\n",
      "Epoch 00010: LearningRateScheduler reducing learning rate to 0.01001953125.\n",
      "Epoch 10/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3274 - acc: 0.8493 - val_loss: 0.3233 - val_acc: 0.8483\n",
      "\n",
      "Epoch 00011: LearningRateScheduler reducing learning rate to 0.010009765625.\n",
      "Epoch 11/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3268 - acc: 0.8485 - val_loss: 0.3315 - val_acc: 0.8511\n",
      "\n",
      "Epoch 00012: LearningRateScheduler reducing learning rate to 0.010004882812500001.\n",
      "Epoch 12/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3258 - acc: 0.8500 - val_loss: 0.3328 - val_acc: 0.8502\n",
      "\n",
      "Epoch 00013: LearningRateScheduler reducing learning rate to 0.01000244140625.\n",
      "Epoch 13/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3278 - acc: 0.8488 - val_loss: 0.3196 - val_acc: 0.8536\n",
      "\n",
      "Epoch 00014: LearningRateScheduler reducing learning rate to 0.010001220703125.\n",
      "Epoch 14/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3261 - acc: 0.8508 - val_loss: 0.3355 - val_acc: 0.8384\n",
      "\n",
      "Epoch 00015: LearningRateScheduler reducing learning rate to 0.0100006103515625.\n",
      "Epoch 15/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3249 - acc: 0.8508 - val_loss: 0.3379 - val_acc: 0.8478\n",
      "\n",
      "Epoch 00016: LearningRateScheduler reducing learning rate to 0.01000030517578125.\n",
      "Epoch 16/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3277 - acc: 0.8485 - val_loss: 0.3253 - val_acc: 0.8524\n",
      "\n",
      "Epoch 00017: LearningRateScheduler reducing learning rate to 0.010000152587890625.\n",
      "Epoch 17/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3251 - acc: 0.8521 - val_loss: 0.3261 - val_acc: 0.8512\n",
      "\n",
      "Epoch 00018: LearningRateScheduler reducing learning rate to 0.010000076293945313.\n",
      "Epoch 18/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3287 - acc: 0.8494 - val_loss: 0.3232 - val_acc: 0.8543\n",
      "\n",
      "Epoch 00019: LearningRateScheduler reducing learning rate to 0.010000038146972657.\n",
      "Epoch 19/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3239 - acc: 0.8512 - val_loss: 0.3334 - val_acc: 0.8496\n",
      "\n",
      "Epoch 00020: LearningRateScheduler reducing learning rate to 0.010000019073486329.\n",
      "Epoch 20/20\n",
      "254/254 [==============================] - 1s 4ms/step - loss: 0.3279 - acc: 0.8504 - val_loss: 0.3174 - val_acc: 0.8523\n"
     ]
    }
   ],
   "source": [
    "history = keras_model.fit(training_dataset, \n",
    "                          epochs=NUM_EPOCHS, \n",
    "                          steps_per_epoch=int(num_train_examples/BATCH_SIZE), \n",
    "                          validation_data=validation_dataset, \n",
    "                          validation_steps=1, \n",
    "                          callbacks=[lr_decay_cb, tensorboard_cb],\n",
    "                          verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fW7vTPm2pd2l"
   },
   "source": [
    "### Visualize training and export the trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S9ZQcgetLHz9"
   },
   "source": [
    "#### Visualize training\n",
    "\n",
    "Import `matplotlib` to visualize how the model learned over the training period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "ISzcM0lxMOPX",
    "outputId": "6a3b5418-2c20-4a29-b3eb-1d0b52cf22c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.0.3)\n",
      "Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.14.6)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.5.3)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.0.1)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.10.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib) (1.11.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib) (40.8.0)\n"
     ]
    }
   ],
   "source": [
    "! pip install matplotlib\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e6tLXXT0SR-q"
   },
   "source": [
    "Plot the model's loss (binary cross-entropy) and accuracy, as measured at the\n",
    "end of each training epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 573
    },
    "colab_type": "code",
    "id": "yU1TzdlY0i-Y",
    "outputId": "f895d4cc-58bf-4ef2-d7d5-85586b0fcdb1"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VfWd//HXJzd7boDADftO0SK4\ngJFaKe5aalu0rtg6o3asU0d0qp3+qjOOtrRO7TLW6QyttZZOF5VarJVOsaitaLUFCYoo4AKIEtaw\nB7Inn98f5yRcQpab5SaB+34+Hvdxz34+9yb3fu453/P9HHN3REREWpPW0wGIiEjvp2QhIiJtUrIQ\nEZE2KVmIiEiblCxERKRNShYiItImJQuRXsbMlpjZDQku62b2oc5uR6QtShbSq5nZRjM7P258lpnt\nMbOzejIukVSjZCFHDTO7FpgLfNLdX2jnumZm+n8X6SB9eOSoYGb/CPwn8HF3/2vc9NPN7K9mttfM\nXjezs+PmLTGze83sZaAcGGtm15vZWjMrM7MN4XYblo+Z2f+F29ptZn9pKcGEp3/+yczeDbf1DTMb\nF8ay38weN7PMuOW/YGbrwu0uNLOhcfMuMLO3zGyfmf0PYE329fkw5j1mttjMRnXg/Uszs7vM7H0z\n22FmvzCzvuG8bDP7lZntCl/7cjMbFM67LnyfyszsPTP7XHv3LccId9dDj177ADYCTwDbgZObzBsG\n7AIuIvjhc0E4XhjOXwJ8AEwE0oEM4JPAOIIv5LMIksiUcPlvAQ+Gy2UA0wFrIS4HngL6hNuvAv4E\njAX6AmuAa8NlzwV2AlOALOC/gRfDeTGgDLg83OdtQC1wQzj/YmAdMCF8DXcBf20Sx4daiHFJ3HY+\nH25nLBAFfgv8Mpz3j8DvgVwgApwavq48YD9wfLjcEGBiT/9P6NEzDx1ZyNHgAmAp8EaT6dcAi9x9\nkbvXu/uzQDFB8mjwv+6+2t1r3b3G3f/g7us98ALwDEFSAKgh+EIcFS77F3dvrXjad9x9v7uvBt4E\nnnH3De6+D3gamBwu9zlgnru/6u5VwJ3AR81sdBjrandf4O41wAPAtrh9fBH4lruvdfda4D+AUzpw\ndPE54P4wvgNhDLPMLD183QMIkk6du69w9/3hevXAJDPLcfet4WuVFKRkIUeDm4DjgIfNLP4UzSjg\nivDUyV4z2wt8jOALv8Gm+A2Z2SfMbGl4OmgvwZd1LJz9XYJf38+Ep17uaCOu7XHDFc2MR8PhocD7\nDTPCL+tdBEdGQ+NjDJNTfMyjgP+Ke327CY6KhrURW1OHxRAOpwODgF8Ci4H5ZrbFzL5jZhnufhC4\niiBhbTWzP5jZh9u5XzlGKFnI0WA7cB7BEcAP46ZvIjiV0i/ukefu98Ut03hkYGZZBKe0vgcMcvd+\nwCLCNgJ3L3P3L7v7WGAmcLuZndcF8W8h+NJviCOP4Jf8ZmArMCJunsWPh6/xH5u8xhyPa7fpSAzA\nSILTXdvDo6ivu/sJwBnAp4C/B3D3xe5+AUECfgv4STv3K8cIJQs5Krj7FoKEMcPMvh9O/hXwaTP7\nuJlFwobas81seAubySRoMygFas3sE8CFDTPN7FNm9qHwC3sfUEdwGqazHgOuN7NTwoT1H8Ayd98I\n/AGYaGaXhqeEbgUGx637IHCnmU0MY+xrZld0MIbbzGyMmUXDGH7t7rVmdo6ZnWhmEYI2ihqg3swG\nmdnFYXKrAg7QNe+HHIWULOSo4e4fEDQWX25m33L3TQQNwP9KkAA2AV+hhf9rdy8j+DJ+HNgDfBZY\nGLfIeOA5gi/FvwE/dPfnuyDu54B/Jziq2UrQwD4rnLcTuAK4j+DU1Hjg5bh1nwS+TXCKaD9B28gn\nOhDGPILTTS8C7wGVwC3hvMHAAoJEsRZ4IVw2Dbid4KhkN8EFATd1YN9yDLDW2+9ERER0ZCEiIglQ\nshARkTYpWYiISJuULEREpE3pPR1AV4nFYj569OieDkNE5KiyYsWKne5e2NZyx0yyGD16NMXFxT0d\nhojIUcXM3m97KZ2GEhGRBChZiIhIm5QsRESkTcdMm4WIHFtqamooKSmhsrKyp0M5JmRnZzN8+HAy\nMjI6tH5Sk4WZzQD+i+CGKg83qQZKWBDunHA0FxgYVgJtuIXmXeG8b7r7z5MZq4j0LiUlJeTn5zN6\n9GgOr0wv7eXu7Nq1i5KSEsaMGdOhbSQtWYQVLOcS3LimBFhuZgvdfU3DMu5+W9zytxDeLMbM+gP3\nAEUEJaZXhOvuSVa8ItK7VFZWKlF0ETNjwIABlJaWdngbyWyzmAqsC+/MVQ3MJ6gQ2pKrCcooA3wc\neNbdd4cJ4llgRhJjFZFeSImi63T2vUxmshjG4Xf8KqGFu3uFt4gcA/y5vet21v7KGr7/7Dus3LQ3\nGZsXETkm9JaroWYBC9y9rj0rmdmNZlZsZsUdPbxyh//607sUb9zdofVF5Ni0d+9efvjDH7a9YBMX\nXXQRe/e2/uPz7rvv5rnnnutoaD0imcliM4ffHnJ4OK05szh0Cirhdd39IXcvcveiwsI2e6s3q092\nOpmRNEoPVHVofRE5NrWULGpra1tdb9GiRfTr16/VZebMmcP555/fqfi6WzKTxXJgfHgbx0yChLCw\n6ULhDeALCO5M1mAxcKGZFZhZAcGtLxcnI0gzIxbNZGdZdTI2LyJHqTvuuIP169dzyimncNpppzF9\n+nRmzpzJCSecAMAll1zCqaeeysSJE3nooYca1xs9ejQ7d+5k48aNTJgwgS984QtMnDiRCy+8kIqK\nCgCuu+46FixY0Lj8Pffcw5QpUzjxxBN56623ACgtLeWCCy5g4sSJ3HDDDYwaNYqdO3d287twSNKu\nhgrv7Tub4Es+Asxz99VmNgcodveGxDELmO9xt+xz991m9g2ChAMwx92Tdp4olp/FTh1ZiPRaX//9\natZs2d+l2zxhaB/u+fTEFuffd999vPnmm6xcuZIlS5bwyU9+kjfffLPx0tN58+bRv39/KioqOO20\n07jssssYMGDAYdt49913eeyxx/jJT37ClVdeyRNPPME111xzxL5isRivvvoqP/zhD/ne977Hww8/\nzNe//nXOPfdc7rzzTv74xz/y05/+tEtff3sltZ+Fuy8CFjWZdneT8a+1sO48gvsGJ11hNItt+9Xx\nR0RaNnXq1MP6KPzgBz/gySefBGDTpk28++67RySLMWPGcMoppwBw6qmnsnHjxma3femllzYu89vf\n/haAl156qXH7M2bMoKCgoEtfT3upBzcQi2bxxuZ9PR2GiLSgtSOA7pKXl9c4vGTJEp577jn+9re/\nkZuby9lnn91sT/OsrKzG4Ugk0ngaqqXlIpFIm20iPaW3XA3Vo2L5mew6WE19vbe9sIikhPz8fMrK\nypqdt2/fPgoKCsjNzeWtt95i6dKlXb7/adOm8fjjjwPwzDPPsGdPz/ZJ1pEFwZFFXb2zt6KG/nmZ\nPR2OiPQCAwYMYNq0aUyaNImcnBwGDRrUOG/GjBk8+OCDTJgwgeOPP57TTz+9y/d/zz33cPXVV/PL\nX/6Sj370owwePJj8/Pwu30+iLK5d+ahWVFTkHb350e9f38Itj73GM7edyXGDeu6PISKHrF27lgkT\nJvR0GD2mqqqKSCRCeno6f/vb37jppptYuXJlp7bZ3HtqZivcvaitdXVkQXBkAbCzrErJQkR6hQ8+\n+IArr7yS+vp6MjMz+clPftKj8ShZAIX5QbJQxzwR6S3Gjx/Pa6+91tNhNFIDN8GlswClZUoWIiLN\nUbIA+uQEJT92HlAvbhGR5ihZENZ6j2aqF7eISAuULEKxqEp+iIi0RMkiFNORhYh0QjQaBWDLli1c\nfvnlzS5z9tln09Yl/g888ADl5eWN44mUPO8OShahwvwsNXCLSKcNHTq0saJsRzRNFomUPO8OShah\nWDSLXQdU8kNEAnfccQdz585tHP/a177GN7/5Tc4777zGcuJPPfXUEett3LiRSZMmAVBRUcGsWbOY\nMGECn/nMZw6rDXXTTTdRVFTExIkTueeee4CgOOGWLVs455xzOOecc4BDJc8B7r//fiZNmsSkSZN4\n4IEHGvfXUin0rqR+FqFYNIvaemdfRQ0FKvkh0rs8fQdse6Nrtzn4RPjEfS3Ovuqqq/jSl77EzTff\nDMDjjz/O4sWLufXWW+nTpw87d+7k9NNPZ+bMmS3e3/pHP/oRubm5rF27llWrVjFlypTGeffeey/9\n+/enrq6O8847j1WrVnHrrbdy//338/zzzxOLxQ7b1ooVK/jZz37GsmXLcHc+8pGPcNZZZ1FQUJBw\nKfTO0JFFKBZ2zFO7hYgATJ48mR07drBlyxZef/11CgoKGDx4MP/6r//KSSedxPnnn8/mzZvZvn17\ni9t48cUXG7+0TzrpJE466aTGeY8//jhTpkxh8uTJrF69mjVr1rQaz0svvcRnPvMZ8vLyiEajXHrp\npfzlL38BEi+F3hk6sgjFosHRROmBKsar5IdI79LKEUAyXXHFFSxYsIBt27Zx1VVX8cgjj1BaWsqK\nFSvIyMhg9OjRzZYmb8t7773H9773PZYvX05BQQHXXXddh7bTINFS6J2hI4vQwMYjC3XME5HAVVdd\nxfz581mwYAFXXHEF+/btY+DAgWRkZPD888/z/vvvt7r+mWeeyaOPPgrAm2++yapVqwDYv38/eXl5\n9O3bl+3bt/P00083rtNSafTp06fzu9/9jvLycg4ePMiTTz7J9OnTu/DVti6pRxZmNgP4L4Lbqj7s\n7kf8PDCzK4GvAQ687u6fDafXAQ0nKT9w95nJjDWmkh8i0sTEiRMpKytj2LBhDBkyhM997nN8+tOf\n5sQTT6SoqIgPf/jDra5/0003cf311zNhwgQmTJjAqaeeCsDJJ5/M5MmT+fCHP8yIESOYNm1a4zo3\n3ngjM2bMYOjQoTz//PON06dMmcJ1113H1KlTAbjhhhuYPHlyUk45NSdpJcrNLAK8A1wAlBDcT/tq\nd18Tt8x44HHgXHffY2YD3X1HOO+Au0cT3V9nSpQDuDvH3fU0N0wfy1dntP4PICLJl+olypOhMyXK\nk3kaaiqwzt03uHs1MB+4uMkyXwDmuvsegIZE0RPMjAF5WezUkYWIyBGSmSyGAZvixkvCafGOA44z\ns5fNbGl42qpBtpkVh9MvaW4HZnZjuExxaWlppwOO5asXt4hIc3r6aqh0YDxwNjAceNHMTnT3vcAo\nd99sZmOBP5vZG+6+Pn5ld38IeAiC01CdDSaoD6UGbpHewt1b7MMg7dPZJodkHllsBkbEjQ8Pp8Ur\nARa6e427v0fQxjEewN03h88bgCXA5CTGCgT3tdCRhUjvkJ2dza5duzr9JSdBoti1axfZ2dkd3kYy\njyyWA+PNbAxBkpgFfLbJMr8DrgZ+ZmYxgtNSG8ysACh396pw+jTgO0mMFQg65u08UKVfMyK9wPDh\nwykpKaErTjFLkHyHDx/e4fWTlizcvdbMZgOLCS6dnefuq81sDlDs7gvDeRea2RqgDviKu+8yszOA\nH5tZPcHRz33xV1ElSyyaRU1dUPKjX65Kfoj0pIyMDMaMGdPTYUgoqW0W7r4IWNRk2t1xww7cHj7i\nl/krcGIyY2tOQy/unQeqlCxEROKoB3ecQ/fiViO3iEg8JYs4KiYoItI8JYs4hSr5ISLSLCWLOH1z\nMkhPMx1ZiIg0oWQRJy3NGKB7cYuIHEHJogn14hYROZKSRRMx9eIWETmCkkUThfmqPCsi0pSSRRMN\np6FUj0ZE5BAliyZi0Uyq6+rZX1Hb06GIiPQaShZNFIYd80rVbiEi0kjJoomGe3GrkVtE5BAliyaU\nLEREjqRk0UTDaShdESUicoiSRRP9cjKIpJnaLERE4ihZNJGWZgzIy2SnypSLiDRSsmiGenGLiBwu\nqcnCzGaY2dtmts7M7mhhmSvNbI2ZrTazR+OmX2tm74aPa5MZZ1MN9+IWEZFA0m6ramYRYC5wAVAC\nLDezhfH30jaz8cCdwDR332NmA8Pp/YF7gCLAgRXhunuSFW+8wmgW63cc6I5diYgcFZJ5ZDEVWOfu\nG9y9GpgPXNxkmS8AcxuSgLvvCKd/HHjW3XeH854FZiQx1sPE8jMpLatSyQ8RkVAyk8UwYFPceEk4\nLd5xwHFm9rKZLTWzGe1YFzO70cyKzay4tLS0ywIvjGYFJT8qVfJDRAR6voE7HRgPnA1cDfzEzPol\nurK7P+TuRe5eVFhY2GVBqWOeiMjhkpksNgMj4saHh9PilQAL3b3G3d8D3iFIHomsmzSNyUId80RE\ngOQmi+XAeDMbY2aZwCxgYZNlfkdwVIGZxQhOS20AFgMXmlmBmRUAF4bTukUsPxNAd8wTEQkl7Woo\nd681s9kEX/IRYJ67rzazOUCxuy/kUFJYA9QBX3H3XQBm9g2ChAMwx913JyvWpgp1GkpE5DBJSxYA\n7r4IWNRk2t1xww7cHj6arjsPmJfM+FpSkJsZlPzQaSgREaDnG7h7pbQ0o39epo4sRERCShYtUMkP\nEZFDlCxaEItmUqoGbhERQMmiRYXRLF06KyISUrJoQWFYTFAlP0RElCxaFItmUVVbT1mVSn6IiChZ\ntKCxY55ORYmIKFm05FB9KDVyi4goWbRAxQRFRA5RsmhBYb6ShYhIAyWLFhTkZpJmqOSHiAhKFi2K\npBn989SLW0QElCxaFYtmUlqmBm4RESWLVjR0zBMRSXVKFq1QMUERkYCSRStU8kNEJKBk0YpYNJPK\nmnoOqOSHiKS4pCYLM5thZm+b2Tozu6OZ+deZWamZrQwfN8TNq4ub3vTe3d1CvbhFRAJJu62qmUWA\nucAFQAmw3MwWuvuaJov+2t1nN7OJCnc/JVnxJSK+F/eYWF5PhiIi0qOSeWQxFVjn7hvcvRqYD1yc\nxP11ucZkoY55IpLikpkshgGb4sZLwmlNXWZmq8xsgZmNiJuebWbFZrbUzC5pbgdmdmO4THFpaWkX\nhh5orDyrK6JEJMX1dAP374HR7n4S8Czw87h5o9y9CPgs8ICZjWu6srs/5O5F7l5UWFjY5cENyMsK\nSn6ozUJEUlwyk8VmIP5IYXg4rZG773L3hp/tDwOnxs3bHD5vAJYAk5MYa7OCkh+Zqg8lIikvmcli\nOTDezMaYWSYwCzjsqiYzGxI3OhNYG04vMLOscDgGTAOaNox3C3XMExFJ4tVQ7l5rZrOBxUAEmOfu\nq81sDlDs7guBW81sJlAL7AauC1efAPzYzOoJEtp9zVxF1S2ULEREkpgsANx9EbCoybS744bvBO5s\nZr2/AicmM7ZExaKZvP/BwZ4OQ0SkR/V0A3evV5ifxc6yapX8EJGUpmTRhlg0i4qaOg5W1/V0KCIi\nPUbJog3qmCciomTRppjuxS0iomTRllhUvbhFRJQs2lAYnoZSL24RSWVKFm3on5eJmdosRCS1KVm0\nIT2SRv/cTEp1GkpEUpiSRQJi0SwdWYhISlOySEAsP1MN3CKS0hJKFmb2z2bWxwI/NbNXzezCZAfX\nWwT1odTALSKpK9Eji8+7+37gQqAA+DvgvqRF1csUqpigiKS4RJOFhc8XAb9099Vx0455sfwsyqvr\nOFhV29OhiIj0iESTxQoze4YgWSw2s3ygPnlh9S6NJT90dCEiKSrREuX/AJwCbHD3cjPrD1yfvLB6\nl/he3KMG5PVwNCIi3S/RI4uPAm+7+14zuwa4C9iXvLB6l4Yji9IyNXKLSGpKNFn8CCg3s5OBLwPr\ngV8kLapeplDFBEUkxSWaLGo9uPvPxcD/uPtcIL+tlcxshpm9bWbrzOyOZuZfZ2alZrYyfNwQN+9a\nM3s3fFyb6AtKhsaSH0oWIpKiEm2zKDOzOwkumZ1uZmlARmsrmFkEmAtcAJQAy81sYTP30v61u89u\nsm5/4B6gCHCCBvaF7r4nwXi7VEYkjYLcTErVi1tEUlSiRxZXAVUE/S22AcOB77axzlRgnbtvcPdq\nYD7BkUkiPg486+67wwTxLDAjwXWTIhZVL24RSV0JJYswQTwC9DWzTwGV7t5Wm8UwYFPceEk4ranL\nzGyVmS0wsxHtWdfMbjSzYjMrLi0tTeSldJh6cYtIKku03MeVwCvAFcCVwDIzu7wL9v97YLS7n0Rw\n9PDz9qzs7g+5e5G7FxUWFnZBOC2LqRe3iKSwRNss/g04zd13AJhZIfAcsKCVdTYDI+LGh4fTGrn7\nrrjRh4HvxK17dpN1lyQYa1Ko8qyIpLJE2yzSGhJFaFcC6y4HxpvZGDPLBGYBC+MXMLMhcaMzgbXh\n8GLgQjMrMLMCgppUixOMNSkK87M4WF1HebVKfohI6kn0yOKPZrYYeCwcvwpY1NoK7l5rZrMJvuQj\nwDx3X21mc4Bid18I3GpmM4FaYDdwXbjubjP7BkHCAZjj7rvb8bq6XGMv7rJqRg5I9G0TETk2JPSt\n5+5fMbPLgGnhpIfc/ckE1ltEk6Ti7nfHDd8J3NnCuvOAeYnE1x1i+Q334q5i5IDcHo5GRKR7JfwT\n2d2fAJ5IYiy9WqGKCYpICms1WZhZGUGnuCNmAe7ufZISVS+kyrMikspaTRbu3mZJj1QxIK7NQkQk\n1ege3AkKSn5k6MhCRFKSkkU7xKJZqg8lIilJyaId1ItbRFKVkkU7xPKVLEQkNSlZtENQeVYN3CKS\nepQs2qEwP4sDVbVUVNf1dCgiIt1KyaId1NdCRFKVkkU7NPTiLlWyEJEUo2TRDo1HFrp8VkRSjJJF\nO8Tyw17cauQWkRSjZNEOA/LUZiEiqUnJoh0y09Pop5IfIpKClCzaSSU/RCQVKVm0U9AxT8lCRFJL\nUpOFmc0ws7fNbJ2Z3dHKcpeZmZtZUTg+2swqzGxl+HgwmXG2R1AfSg3cIpJaknYzaTOLAHOBC4AS\nYLmZLXT3NU2Wywf+GVjWZBPr3f2UZMXXUbFoli6dFZGUk8wji6nAOnff4O7VwHzg4maW+wbwbaAy\nibF0mcL8LMqqaqmsUckPEUkdyUwWw4BNceMl4bRGZjYFGOHuf2hm/TFm9pqZvWBm05vbgZndaGbF\nZlZcWlraZYG3RvfiFpFU1GMN3GaWBtwPfLmZ2VuBke4+GbgdeNTMjrjft7s/5O5F7l5UWFiY3IBD\nDR3zdEWUiKSSZCaLzcCIuPHh4bQG+cAkYImZbQROBxaaWZG7V7n7LgB3XwGsB45LYqwJO1RMUI3c\nIpI6kpkslgPjzWyMmWUCs4CFDTPdfZ+7x9x9tLuPBpYCM9292MwKwwZyzGwsMB7YkMRYE6bKsyKS\nipJ2NZS715rZbGAxEAHmuftqM5sDFLv7wlZWPxOYY2Y1QD3wRXffnaxY22NANKwPpdNQIpJCkpYs\nANx9EbCoybS7W1j27LjhJ4AnkhlbR2WlR+ibo5IfIpJa1IO7A2LRTN3TQkRSipJFBwQd89TALSKp\nQ8miA2L5WToNJSIpRcmiAwqjWToNJSIpRcmiA2LRTMoqVfJDRFKHkkUHFOYHfS12HVS7hYikBiWL\nDmjomKeSHyKSKpQsOqCxF7eShYikCCWLDojlq+SHiKQWJYsOGJAXlvxQshCRFKFk0QHZGRH6ZKer\n8qyIpAwliw6K5auvhYikDiWLDopFs3Q1lIikDCWLDiqMquSHiKQOJYsOikUzdemsiKQMJYsOikWz\n2F9ZS1WtSn6IyLFPyaKDGkt+6IooEUkBSU0WZjbDzN42s3Vmdkcry11mZm5mRXHT7gzXe9vMPp7M\nODtCJT9EJJUk7baqZhYB5gIXACXAcjNb6O5rmiyXD/wzsCxu2gnALGAiMBR4zsyOc/dec85HvbhF\nJJUk88hiKrDO3Te4ezUwH7i4meW+AXwbqIybdjEw392r3P09YF24vV4jFlUvbhFJHclMFsOATXHj\nJeG0RmY2BRjh7n9o77rh+jeaWbGZFZeWlnZN1AlqLCaoNgsRSQE91sBtZmnA/cCXO7oNd3/I3Yvc\nvaiwsLDrgktAdkaE/Ox0tVmISEpIWpsFsBkYETc+PJzWIB+YBCwxM4DBwEIzm5nAur2COuaJSKpI\n5pHFcmC8mY0xs0yCBuuFDTPdfZ+7x9x9tLuPBpYCM929OFxulpllmdkYYDzwShJj7RCV/BCRVJG0\nZOHutcBsYDGwFnjc3Veb2Zzw6KG1dVcDjwNrgD8CNyftSqj6eihZAfu3QH37dhHLz9SRhYikhGSe\nhsLdFwGLmky7u4Vlz24yfi9wb9KCa1CxGx4+Nxi2NIgOhj5DIH8I9BkaPPKHhtPC58w8IDiyePnA\nrqSHKCLS05KaLI4KmXlw9fzgyKJsK+zfCvs3w6518N5foGrfketk94X8ofxDTV8m1mRT++dXST/9\nRsjt3/3xi4h0AyWLjBw4/hMtz686ECaRLYeew+Ho1o2cFXmbyIsvQOlquOqX3Ra2iEh3UrJoS1YU\nssZDbPwRs15ds50v/KKYZWcUM+jV++GDpTDy9B4IUkQkuVRIsBMaenGvHfv3QVvHM3eBew9HJSLS\n9ZQsOqGhF/eOinQ49y4oWQ5rftfDUYmIdD0li05oKFNeeqAKTvksDJwIz30NanU5rYgcW5QsOiE7\nI0I0Kz3oa5EWgQu/AXs2wvKHezo0EZEupWTRSYX5WYeKCX7oPBh3LrzwHSjf3bOBiYh0ISWLTopF\nMykti6uufsE3oHIf/OU/ey4oEZEupmTRSbFo1uFlygdPgsmfg2U/ht3v9VxgIiJdSMmik2LNVZ49\n598gkgF/mtMzQYmIdDEli06KRbPYW15DTV39oYl9hsIZt8Dq38Km5T0XnIhIF1Gy6KSGy2d3Nb1j\n3hm3Qt5AddQTkWOCkkUntXgv7qwonPOvsGkprP198gJwh9VPBvWqRESSRMmik2INHfOauwnS5L+D\nwg/Dc/dAbRLu1e0Oi/8NfnMdzJuhhCEiSaNk0UmFYcmP1Vv24U1PN0XSg0tpd2+A4nldu2N3+OOd\nsHQuTLo86Nfx85lwYEfX7kdEBCWLThvcN5vjB+XzvWfe4dIf/ZXn395xeNIYfwGMOQteuA8q9nbN\nTt3h6a/Csh/B6TfDZQ/D5x6HfSXwi0vUIVBEulxSk4WZzTCzt81snZnd0cz8L5rZG2a20sxeMrMT\nwumjzawinL7SzB5MZpydkRHDiutNAAAXVElEQVRJY+Et07j3M5PYsb+K63+2nIvnvsxza7YHScMM\nLvxmkCheur/zO3SHRV+BV34MH50NH7832MeoM+DqR2HXu/Cry6Byf+f3JSISsiNOnXTVhs0iwDvA\nBUAJsBy42t3XxC3Tx933h8MzgX9y9xlmNhr4P3eflOj+ioqKvLi4uAtfQftV19bz5Gsl/M/z69i0\nu4KJQ/twy7njufCEQaQ99U/w5hMwezkUjOrYDurrYdG/QPFPg6utLpgTJIp4bz8Nv74Ghp8G1zzR\neAtYEZHmmNkKdy9qa7lkHllMBda5+wZ3rwbmAxfHL9CQKEJ5wFF9jWlmehpXnTaSP3/5bL57+Ukc\nrKrli79awUU/+At/HnojbgZ//kbHNl5fD4u+HCSKaV9qPlFAcNe/yx6GTctg/mehpvLIZURE2imZ\nyWIYsCluvCScdhgzu9nM1gPfAW6NmzXGzF4zsxfMbHpzOzCzG82s2MyKS0tLuzL2TsmIpHFF0Qie\nu/0svn/VyVTX1fP5J7fwWNqn4Y3fULdpRfs2WF8Pf7gtaCT/2O1w/teaTxQNJn4GLp4LG5bAb65N\nzpVYIpJSeryB293nuvs44KvAXeHkrcBId58M3A48amZ9mln3IXcvcveiwsLC7gs6QemRND4zeTjP\n3nYWP7h6Mr/Ouoyd3oc3//cWnnx1E7Xxvb5bUl8P//fPsOJ/Yfq/wHl3t54oGpzyWfjk/fDOH+G3\nX4C62k6/HhFJXclMFpuBEXHjw8NpLZkPXALg7lXuviscXgGsB45LUpxJF0kzZp48lCdvm8G2ybdx\nct1qFi2Yx/n3v8BvijcdXiokXn09/P4WePUXcOZXgrvxJZIoGpz2D3DhvcHd+566OdieiEgHJDNZ\nLAfGm9kYM8sEZgEL4xcws/Fxo58E3g2nF4YN5JjZWGA8sCGJsXaLtDRj0qdvwWPH8UD/39InE76y\nYBXn/ucSvrVoLX98cxs7Gsqd19fBwtnw2q/grK8GxQnbkyganDEbzrkLVs2HP9yu0iMi0iHpydqw\nu9ea2WxgMRAB5rn7ajObAxS7+0JgtpmdD9QAe4Brw9XPBOaYWQ1QD3zR3Y+NzgORDOyCOeQ9Noun\nLlrHn/Nn8uMXNzDv5feoeTHIhyP7ZfLdzJ/wkf2L2Tb5NgaceQcZHUkUDc78F6g5CC99HzJyD11u\nKyKSoKRdOtvdesOlswlzh59/GnasgVtfg+y+VNbUsXrLfla+v5NJy+/kI2XPcn/N5fyg7lKy0tM4\naXhfpowsYPLIAqaM6sfA/Oz27/OPd8CyBw+d0hLpbtXlUL4LKnYHnUcbngeMgzFnQ1qPN6OmnEQv\nnU3akYW0wiy4X/dDZ8NLD8D595CdEeHU4fmcWvz/oOxZOOcuZp08m+M+2MOr7+/l1Q/2HHb0Maxf\nDlNGFTBlZD+mjCzgQwOj5GW18uc0g49/C2rK4cXvQkYOTP9y97xeObbt3QQ71oZf/LsOTwLlu6Bi\nz6Fpta1cyl0wGk69HiZfA3mxbgtfEqMji570xBdg7UKYXQz5Q+DJf4Q3F8C5/x6cOmqi4ejjtQ/2\n8GqYRLbtP/ThK8jNYHhBLsMLchhekMOI/g3DuQzrlxMkk/q6YD9v/AZm3Aen39Sdr1iOJXs3wYvf\ngdceAa87NN3SIKcAcgdATn/I7X/o+bDhcH5OP9j4EhT/DN5/CdIy4ISLoejzQWWCY/2UaU1l8J6l\nZ/bI7hM9slCy6El7P4D/LoIJnwY86OF93j0w/faEN7FlbwUrN+3l/V3llOwpp2RPReNzVe3hVz/1\nz8tkeEEOI/tlcsuub3L8nhdYe9o3iRRdx4C8TPrkZJARaeE0QH097N8cFEXcvQH2vBcOb4Q9GyEz\nF/oMg77DoM/w4AZQDcN9h0F0cFBYUY5UvhsObIfqg8GjpjzuuTxob6ouD8cPxA2Hy9RUwOATYeqN\nMGxK8uMt2x7cY37Fz4LxU6+HEy8Pvvxz+0NW346fTip9O0garz8a3Ms+djwUXQ8nzwoS0LGi6gC8\n+0xwpeK7zwZH+h+7DU67IRjuRkoWR4tn74GXHwiGz/86fOxLXbJZd6f0QFWYPCriEkkwvH3Pfv7H\nvsdZaau4reYmnqr/GOnUMj5zNxMydzIuo5Qxtp3hbGVQ7VYG1Gwl3Wsat1+flkF1/gjq+o0hrf9o\n0morSSvbjO3fQlrZZtJqDh4ej0WoyxtEbXQIddGh1ESHUpM3hNroEGryhlLfbxTp0RhZGelkpqeR\nFT7sWPxVWV0OH/wVNrwQdJzctqrtdSwNMvKCpJyRG5RxycgNxiNZ8P7LQSIZPhU+8o/BL/NIRtfG\nXb47+F9d9hDUVQeni878CvQb0fa67VVdHtynpXgebC6G9GyYdFlwtDHs1KPzaKOqDN5ZHCaI56C2\nIrhB2oRPBT+41v85+FF15r/AlGu77UhDyeJoUbkPfnU5TLq0W08J1dc7O/fuJec3V5O37RUOZg8m\nr2IbaRw6nVBp2WyxwbzPYDbUFrKubiAbfTAf1A9kKwOob/HKa6cP5QyxXQyx3eHzLobabobQMLyL\nbKs5bK0yz2GTD+R9H8j7PohNPpDNNphtkSHsjgwkPSMjTCSRxoTS8JyTGSEnI53czAi5mRFyGp/T\nyc0IhnOzgvk5DeOZ6Y3LpacFXz7xH4eGwfjPSPynJX7ZSJoRSWvhC6yuFrauhA3PBwli07LgyzYt\nA0aeHlQlHjDu8ATQkBgyo8G09KzWvyAr98PKR4MCk7s3BKc1i/4BTr0Ookd2WK2sqWN/ZQ37K2rY\nV1GDmdEnO5387Ayi4fvUmKgr98Hffgh/mxskpJOuDC7nHjCu5Xi60tbXg6ONVY8HR1mDTwySxolX\ncJAcdh6oIs2M/nmZh8fdxdydfRU1bN5bwZa9lWzZW8GWfcFwepoxNpbHuIFRxhVGGTUgl+yMSPB3\naUgQ654L2myig+GEmUFCH/lRSIsEO9j4clAO6IO/Qd+RcPZX4aRZLR6RN8RTsqeCmrp6Jo/s2JGX\nkoUkpuoALL4z+CXXfwz0HwsF4XN04GFfUNW19ewLv1z2VRz6otlXUUO9O0bQl8TMgmEz0izYhJmR\n1jA9Lejgk1Wzj5yKreRWbCP7YAnZZe+Tc7CE6MFNRCs2k+6HypTUEWFvxiBKM4ayI30IW9MGsyVt\nMJsZxAcMYndtFhXVdZRX11JeXXfEKbjuYAYZaWlkROBDaduYlvYGH+ENptS/ST7lAKyPjGVV5mRW\nZ09mXdaJ1GfmkpFmcUdTEbIy4obT08jKSCMzkkZWRuSI6Q2Js6qmjv2VtewrryK/ZAnHb3yE0XuX\nUmsZLMs9h6eyPsVrtaMb/15tvT+RNKMws5a/T3+Ga+p+Rx8vozh3Os8O+jwH+44nmpVBfnZ6Y4Jp\nSNDZGUEyzs6IhAk8eGSlp5HWUjJtory6ltKyKnYeqKK0rJrSA1XsLKuibN9uxm17mml7n2J07Xsc\n9GyerJvGI3Xns9aD4pzZ6TAkL0JhbhoD84zCHBiQY8SyoSAbCjKdgmzomwF9MuvJtjqoq4L0HKpz\nYuzwvpRURdlcVteYDDY3JIa9FZRX1x0Wa2YkjSH9sqmtczbvrQAgn3IuiKzg0uwVfKTuNTKooTxr\nIPvHXET2yZfS9/iPYQ0Joil3WP8n+PM3Yctr1Pcfx5ZTbmNN//PYtLeKkj3lbNodnB3YvKeCsqqg\nMsPJw/vy1OyPJfT+NqVkIUe3+noo2xq2jbwXHKbHD1c06XYTHQSx46DweIgdT92A46joO5byzELK\na+opr66joiZIJOXVdWFiOZRc6uM+B0FKC4etYRpHTAuGg5GsylKG7l7GiD2vMGrfK/SpCWqV7coY\nwjt5p7ImZwprsk5mL32pqXdqauupra+nus6praunuraeqtp6qmrrGocra+qo7+DH0wxOzNrOtZFn\nuKjueXKoZF32JJYVXsGmQeeRn5dDn5wM+uZk0Cc7HXcoq6qlrLKG8oMHGfvBAqaW/Iz82t2syvkI\nj+Zdw5s+hrLK2vBRQ01d+4I7dAR4KKFkZwTTKmvqGxNE0y/kBv3zMolFM4nlZVKUsZ7zDvyBiXv+\nRHp9FbWRHKy+hoh3TVmb3R5lhxewL1JAeeYAanMKITqI9H5DyOs/lD6xYcQGj6T/gIGkRdKgYi9V\nq39P1aonySt5kUh9DXvSC1kSOYPHy6ewtGYcHh6J98lOZ9zAKGNjUcYNzGNcYZSMiDUmgeD5ION2\n/4Uv1j/GhLRNrK0fwf21V/By+lRGFOQdcQHL6FguHx58REWkhChZyLGtYu/hCWTXuqBxdOc7UBVX\nzDirT1wSGR80mBYeH1ym2dKvuwbuwWmXAzuCx8GG59LDp5Vth30fBOvk9IexZ8HYs4PTS/3HdOpl\n1tY1JJHDE0lVTTDeMD0rPUKf7ODLv29OBtHs9EOnxSr3BVcsvfLj4D3LHwqnfT5omI6/RLWuJqgY\n8OJ3g4sZRk8P+uOMPL3Z2Cpr6hoTR3l1HZU1dVTW1FNRU0dFTR2V1XWNwxXVdVTWxk+rD6aF8zMj\naRTmZxGLZoXPmcTysygMx/vnZTZ/8UX57uDKvr0fQCQzOF0XyQiGI8GwRzKp8nQO1KaxvyaNfdXG\n3mrYWwW7K41dlU4O1YzKOsDQ9P0MtH30q99NXvUuIuU7gosPyrYHRyBNRTKDdocD26G+BvqOCE4v\nnXBJ0LaSlkZ9vbN1fyXrdxxgfekBNpQeZH1pMLx9/+HbzEpPOywRjOiXzWkHX2Di2/9D1v738KGT\nsXPvgnHndVm7jZKFpCZ3KNsGO9+G0neC553vBMMHth1aLpIJAz4UJJJYWHbs4A44UBp88BuGayua\n2YkFV/5EB0JeYfA8aBKMOwcGndh7O5bV1wVX3ix7MGg/iWQFVzFN/UKQaJd8K0gmw08LLt8ee1ZP\nR9x7uAc/Qsq2B/8fB7aHPxjC4bzCMEFMadeXeFllDe/tPEhtvTOiIJdYNLP5Npe62qBkz5JvBz9M\nRp4RJPLR0zr90pQsRJqq2As73w0TSUMSeRv2vh98GeTFgl+J0cLwOXwcNm1QkCiO9suAd7wFrzwE\nrz8WXH4LQcPxuf8O4y88Oq82SgW1VUFh0Re/F/z4GXduUPtt+Kkd3qSShUiiaqvAIkd/AuiIir1B\nR9DoIDj+k733qEgOV10e3Ajtpe8HveRPuASu+N8OJXmV+xBJVHpWT0fQc3L6BR3B5OiSmQtn3BJc\nGr30weCS3CQfDSpZiIgcrbLy4ayvdMuudMwpIiJtUrIQEZE2KVmIiEiblCxERKRNSU0WZjbDzN42\ns3Vmdkcz879oZm+Y2Uoze8nMToibd2e43ttm9vFkxikiIq1LWrIwswgwF/gEcAJwdXwyCD3q7ie6\n+ynAd4D7w3VPAGYBE4EZwA/D7YmISA9I5pHFVGCdu29w92pgPnBx/ALuHlfEhzwOVYC+GJjv7lXu\n/h6wLtyeiIj0gGT2sxgGbIobLwE+0nQhM7sZuB3IBM6NW3dpk3WHNbPujcCNACNHjuySoEVE5Eg9\n3inP3ecCc83ss8BdwLXtWPch4CEAMys1s/c7EUoM2NmJ9ZNN8XWO4uscxdc5vTm+UYkslMxksRmI\nv9/i8HBaS+YDP+rgurj7kbcDawczK06kPkpPUXydo/g6R/F1Tm+PLxHJbLNYDow3szFmlknQYL0w\nfgEzGx83+kng3XB4ITDLzLLMbAwwHnglibGKiEgrknZk4e61ZjYbWAxEgHnuvtrM5gDF7r4QmG1m\n5wM1wB7CU1Dhco8Da4Ba4GZ3b/72WSIiknRJbbNw90XAoibT7o4b/udW1r0XuDd50R3hoW7cV0co\nvs5RfJ2j+Dqnt8fXpmPmfhYiIpI8KvchIiJtUrIQEZE2pVSySKBWVZaZ/Tqcv8zMRndjbCPM7Hkz\nW2Nmq83siPYcMzvbzPaFtbRWmtndzW0ryXFujKvndcR9bC3wg/A9XGVmU7oxtuPj3puVZrbfzL7U\nZJlufQ/NbJ6Z7TCzN+Om9TezZ83s3fC5oIV1rw2XedfMEu5/1AXxfdfM3gr/fk+aWb8W1m31fyGJ\n8X3NzDbH/Q0vamHdVj/vSYzv13GxbTSzlS2sm/T3r0u5e0o8CK7IWg+MJegt/jpwQpNl/gl4MBye\nBfy6G+MbAkwJh/OBd5qJ72zg/3r4fdwIxFqZfxHwNGDA6cCyHvx7bwNG9eR7CJwJTAHejJv2HeCO\ncPgO4NvNrNcf2BA+F4TDBd0U34VAejj87ebiS+R/IYnxfQ34lwT+/q1+3pMVX5P5/wnc3VPvX1c+\nUunIos1aVeH4z8PhBcB5Zkm+sW3I3be6+6vhcBmwlmZKnBwFLgZ+4YGlQD8zG9IDcZwHrHf3zvTq\n7zR3fxHY3WRy/P/Zz4FLmln148Cz7r7b3fcAzxIU1Ux6fO7+jLvXhqNLCTrF9ogW3r9EJPJ577TW\n4gu/O64EHuvq/faEVEoWzdWqavpl3LhM+GHZBwzolujihKe/JgPLmpn9UTN73cyeNrOJ3RpYwIFn\nzGxFWJurqUTe5+4wi5Y/pD39Hg5y963h8DZgUDPL9Jb38fMER4rNaet/IZlmh6fJ5rVwGq83vH/T\nge3u/m4L83vy/Wu3VEoWRwUziwJPAF/yw6vyArxKcFrlZOC/gd91d3zAx9x9CkHp+ZvN7MweiKFV\nYcWAmcBvmpndG97DRh6cj+iV16+b2b8RdIp9pIVFeup/4UfAOOAUYCvBqZ7e6GpaP6ro9Z+leKmU\nLBKpN9W4jJmlA32BXd0SXbDPDIJE8Yi7/7bpfHff7+4HwuFFQIaZxborvnC/m8PnHcCTHFk6vt11\nvZLgE8Cr7r696Yze8B4C2xtOzYXPO5pZpkffRzO7DvgU8LkwoR0hgf+FpHD37e5e5+71wE9a2G9P\nv3/pwKXAr1tapqfev45KpWTRZq2qcLzhqpPLgT+39EHpauH5zZ8Ca939/haWGdzQhmJmUwn+ft2Z\nzPLMLL9hmKAh9M0miy0E/j68Kup0YF/cKZfu0uIvup5+D0Px/2fXAk81s8xi4EIzKwhPs1wYTks6\nM5sB/D9gpruXt7BMIv8LyYovvg3sMy3sN5HPezKdD7zl7iXNzezJ96/DerqFvTsfBFfqvENwlcS/\nhdPmEHwoALIJTl2sIyhcOLYbY/sYwemIVcDK8HER8EXgi+Eys4HVBFd2LAXO6Ob3b2y479fDOBre\nw/gYjeAOieuBN4Cibo4xj+DLv2/ctB57DwmS1laC+mclwD8QtIP9iaBw5nNA/3DZIuDhuHU/H/4v\nrgOu78b41hGc72/4P2y4QnAosKi1/4Vuiu+X4f/WKoIEMKRpfOH4EZ/37ogvnP6/Df9zcct2+/vX\nlQ+V+xARkTal0mkoERHpICULERFpk5KFiIi0SclCRETapGQhIiJtUrIQ6QXCarj/19NxiLREyUJE\nRNqkZCHSDmZ2jZm9Et6D4MdmFjGzA2b2fQvuQ/InMysMlz3FzJbG3ReiIJz+ITN7Lixm+KqZjQs3\nHzWzBeG9JB7prorHIolQshBJkJlNAK4Cprn7KUAd8DmCXuPF7j4ReAG4J1zlF8BX3f0kgh7HDdMf\nAeZ6UMzwDIIewBBUGv4ScAJBD99pSX9RIglK7+kARI4i5wGnAsvDH/05BEUA6zlUMO5XwG/NrC/Q\nz91fCKf/HPhNWA9omLs/CeDulQDh9l7xsJZQeHe10cBLyX9ZIm1TshBJnAE/d/c7D5to9u9Nluto\nDZ2quOE69PmUXkSnoUQS9yfgcjMbCI330h5F8Dm6PFzms8BL7r4P2GNm08Ppfwe84MFdEEvM7JJw\nG1lmltutr0KkA/TLRSRB7r7GzO4iuLtZGkGl0ZuBg8DUcN4OgnYNCMqPPxgmgw3A9eH0vwN+bGZz\nwm1c0Y0vQ6RDVHVWpJPM7IC7R3s6DpFk0mkoERFpk44sRESkTTqyEBGRNilZiIhIm5QsRESkTUoW\nIiLSJiULERFp0/8H9+XX7VZlZH8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd8VfX9+PHXO5uELEiYYe8hiCBO\nFDdYtziwtmprbV211v5a2m+raJdtrW1t3a1ara0iLrRYcYALVDay90gCJGSTndz374/PCVzCTXIh\nublJeD8fj/vIved8zjnvXMJ93884n4+oKsYYY0xjIsIdgDHGmLbPkoUxxpgmWbIwxhjTJEsWxhhj\nmmTJwhhjTJMsWRhjjGmSJQtjWpiILBCRm4MsqyIyONQxGdNclixMSInIdhE51+/1tSJSICJnhjMu\nY8yRsWRhWo2I3AA8CnxNVT86wmNFROzvtRWJSGS4YzBth/3nM61CRL4L/BG4QFUX+m0/WUQWikih\niKwUkcl++xaIyK9F5DOgDBgoIjeJyDoRKRGRrd5568qnicjb3rnyReSThhKM1/xzm4hs8s71SxEZ\n5MVSLCKzRCTGr/x3RGSzd945ItLLb995IrJeRIpE5G+A1LvWt7yYC0TkXRHpF+R71uDv6u2/VERW\nePFuEZEp3vYuIvKsiGR713zD236jiHwa4H0Y7D1/TkQeF5G5IlIKnCUiXxOR5d41donIzHrHn+73\n77fLu8aJIrLXP9mIyBUisjKY39u0UapqD3uE7AFsB14F9gJj6+3rDeQBF+K+uJznvU739i8AdgKj\ngCggGvgaMAj3gXwmLomc4JX/LfCEVy4amARIA3Ep8CaQ5J2/EvgAGAgkA2uBG7yyZwP7gBOAWOCv\nwMfevjSgBJjmXfNuoAa42dt/KbAZGOH9Dj8HFtaLY3ADMTb2u04Eirz3LMJ7L4d7+/4LvAykejGd\n6W2/Efg0wPsw2Hv+nHfO07xzxgGTgeO812O8f8fLvPL9vN99unedrsDx3r61wFS/67wO3BPuv0d7\nNOP/crgDsEfHfnjJotj7YI6ot+8nwAv1tr3r9yG9AHigifO/AdzlPX/Au07AD996xylwmt/rpcBP\n/F7/Efiz9/wfwO/99nUGqoH+wDeBz/32CZDplyzeAb7ttz/C+9Dv5xdHk/EG+F2fBP4UoExPwAek\nBtgXTLJ4vokY/lx3XeCnwOsNlPsJ8KL3vIv3O/cM99+jPY7+Yc1QpjXcCgwF/i4i/k00/YCrvCaM\nQhEpBE7HfeDV2eV/IhGZKiKfe81BhbhaSZq3+w+4b/HzvGabGU3EtdfveXmA1529572AHXU7VHU/\nrgbU29u3y2+f1ou5H/AXv98vH5dQejcRW1O/ax9gS4DD+gD5qlrQ1PkbUP/9PklE5otIrogUAd8L\nIgaAfwEXi0gCcDXwiaruPsqYTBtgycK0hr3AObhmocf8tu/C1SxS/B4JqvqgX5kD0yKLSCyuSesh\noLuqpgBz8foIVLVEVe9R1YHAJcAPReScFog/G/ehXxdHAq7JJQvYjfvQrNsn/q+93/G79X7HTurX\nbxNIU7+rd95BAQ7dBXQRkZQA+0qBeL9r9AhQpv401P8G5gB9VDUZ18zXVAyoahawCLgC+AbwQqBy\npv2wZGFahapm4xLGFBH5k7e57tvnBSISKSJxIjJZRDIaOE0Mrs8gF6gRkanA+XU7ReQiERnsfWAX\nAbW4Jpnm+g9wk4gc732I/wb4QlW34/oHRnkduFHA9wH/D+EngJ+KyCgvxmQRuSqIazb6u+Kaxm4S\nkXNEJEJEeovIcO/b+zvAYyKSKiLRInKGd8xKL9bjRSQOmBlEHIm4mkqFiEwErvPb9yJwrohcLSJR\nItJVRI732/888GNcn8drQVzLtGGWLEyrUdWduM7iaSLyW1XdhesA/hnuQ3EX8P9o4O9SVUtwH8az\ngALcB9ccvyJDgPeB/bhvtY+p6vwWiPt94Be4b/q7cd+mr/X27QOuAh7ENU0NAT7zO/Z14HfASyJS\nDKwGpgZxzUZ/V1X9ErgJ+BMuMX7EwdrPN3B9KuuBHOAH3jEbcf067wObgENGRjXgNuABESkB7vXi\nqYthJ65p7B5c89oKYKzfsa97Mb2uqmVBXMu0YeKaWI0xpuWJyBZcM9z74Y7FNI/VLIwxISEiV+L6\nQD4Mdyym+aLCHYAxpuMRkQXASOAbqtoS/UYmzKwZyhhjTJOsGcoYY0yTOkwzVFpamvbv3z/cYRhj\nTLuydOnSfaqa3lS5DpMs+vfvz5IlS8IdhjHGtCsisqPpUtYMZYwxJgiWLIwxxjTJkoUxxpgmWbIw\nxhjTJEsWxhhjmmTJwhhjTJMsWRhjjGmSJQtjjDlaqrDubdjS7Jnw27wOc1OeMca0qj1fwTs/gR2f\nAQJTfwcnfTfcUYWM1SyMOZapwuYPIK+hpbTNYUrz4O274ckzIHc9fO1hGP41eOfHMO8X4OuYk+xa\nzcKYY1VRJvz3Htj4P+h7KnzrnXBH1LbV1sCSZ2D+r6GyBCZ+Fyb/BDqlwvgbXbJY+AiU7IZLH4Wo\n2HBH3KIsWRhzrPH5YMk/4P37QWuh3+mw41OXPJIbWv78GLf1I/jfDMhZCwMnw5QHoduIg/sjIuHC\nhyCpN3xwP5TsgWtfhLjkcEXc4qwZyphjSe4GeHYKzP0RZEyA2xbBJY+4fWteD29sbVHBDnj5G/D8\nJVC1H655Eb7xxqGJoo4ITPohXP4k7FwEz0yF4uzWjzlErGZhzLGgpgo+/RN88hDEJMBlT8DYa90H\nHECvcfDVbDj1zvDGeaR8te73KtgOGSe6R/pwiGjm9+CqMnfehY+ARMDZP4dT7oTouKaPHXstdO7u\nkszfz4XrXw2cXFqCrxZWvwaVxXDit0NzDY8lC2M6ul2LYc6dkLsORl/pmlA6dzu0zOhpMO//XEd3\n10HhifNIVZXBa9+B9W9DbBIsf8Ftj02C3uOhz0TImAgZ412/QgC78su4/6211Ph83Hn2EMb3TYE1\nr8G8e6E4070v5z0Ayb2PLLZBZ8FNc+HFq+AfF8D0f0P/05v5C/vx+WD9WzD/t+7ftc/JMOFbB5N/\nCHSYZVUnTJigtp6FMX4q98OHv4QvnoSkXm7UzrApgcsWZcGfRsJZ/wdn/rh14zwapfvg39dA1lKY\n8ls46Xsu0WV+CZmLXYLMWQN1y3+nDXWJo4+rfWjaMGYty+aBt9YiIsRFR5Beuok/J/2HYZWroMdx\nMPX30O/U5sVZuBP+NQ0KtrnmqdFXNO98qrDxXdfJvmcVdB0CZ/0URl5+1LUpEVmqqhOaLGfJwpgO\naNP78PYPXKf1iTfDOfdCXFLjxzwzFcry4PYvQvoNtdnytsC/rnSjjq78O4y4OHC5yhLIXg676hLI\nl1CeD0C5xLOkZiC5yWM446wpJGd+SOTy5ykigd9XX03JyOn84LwRDO7WufnxluXDS1+HnQvhgt/A\nKbcfsju/tIqSimoyUuOJjGjgfVeFrfPhw19D1hJI7Q9nzoDjroLI5jUQWbIw7Ufd32BrfkDV1rib\nqda+CZveg27D3Qdqj+NaLwZwN3Z98jBUl0P6MNfenj7MfROOPYoPqtI8N2rnq1mQNsx1Xvc9Obhj\nF//dDaW9dSF0H3Xk124NO7+A/3h9LdNfdjWFYKny4aLP+eC9txlVu4ELknfSZf8mRH0gkTDxOxSd\n/CP+vjifZz7dRnl1LVeckMFd5wyhT5f45sVdXQGv3+L+3k6+jZIzZ/Lu2lzmrMzms837qPUpMZER\nDEhLYHC3zgxKT2BQt84MSu/M4PKVxH38W5dskjLgzP8Hx38dIqObF5OnTSQLEZkC/AWIBP6uqg/W\n298X+CeQ4pWZoapzRaQ/sA7Y4BX9XFW/19i1LFm0M75a2PYxrHoZ1r3l2pQHTnZtvQPOhIS0lr9m\nbTVs+8j9h13/X/ctOqoTDDwTdn4OFUXum9rZ/+e+uYVS3haY/xtYPdsNr0zqDfs2ga/6YJnkPocm\nkPThLol0Sjn8fKrw1SsuUVQUu1E5k+45srH+pfvgoaFw2l1w7n1NFi8qq2bOqmxeW5ZJdmE5ESII\nICKI4F7X/cR9vosIEQKCHPJ6RM8kbjilP8dlNDLUdO2b8Notrknt67OPqG+loLSKe+es4a2V2Yzt\nk8LDV49lUHpn11S3ewUk9jzkfHn7K3l8wRae/3wHqsq1J/bljrMH0z0piA7uBlRUVbNn1g/pv/l5\n5vpO5u6q79GtSzKXjO1F3y7xbM0tZUvufjbn7Gdnfhlj2cQPo15hUuRq9pHK3NTr2Nn/Kgb06MKg\ndJdI0jrHIM38khX2ZCEikcBG4DwgE1gMTFfVtX5lngKWq+rjIjISmKuq/b1k8baqjg72epYs2ok9\nX7kE8dVs14wQm+SaESqKYNsnUFnkyvUY4xLHwLPcN+PoTkd3vZpKN2/P2jdhw3/ddWI6w9ApMPIS\nGHyuGx1UXgCf/hm+eMIlshO/DWf8v5ZPWkVZ8NHvYPm/3Af5ybe6EUidUl1tp2Cbuys4dz3kbnQ/\n922EmoqD5+jc49AkktoPPn8CNr8HvSfAJX+F7iOPLr4XroC8zXDXyoA1vVqf8tnmfbyyNJN31+yh\nqsbH8B6JjM1IQVFUwaegqijg07pt7rX6v/bK1vp8fLktn9KqWk7om8INp/Zn6uiexET5tcEvegze\n/Zkb7TT9JUjoGvSvNH99Dj9+dRWFZVXcdc4QvnfmIKIig2vf31NUwV8/3MTLi3cRGSHccGp/vnfm\nILokxAR1fE2tj0Vb85izIpv/rd5DSWU1d8W/y92+5ynpMZHO35yFxNfrfM9eQe2HvyZy8zwqY1JZ\n3PsG3oiewvq8GrbklFJeXXugaHKnaAalJzBxQFdmTB0e9Hviry0ki1OAmap6gff6pwCq+lu/Mk8C\nW1X1d175P6rqqZYsOpiiLPetd9Us1+kYEQWDz4Ox17gP7bpEUFvjvuVtme/aZ3d96b5pR8W5hDHw\nLFf76DGm8c68qjLY8oGXIP4HVSUQmwzDL4SRl7rzNDQEsjgbFjzoRtZEx8Op33dtzEfTJOSvNA8+\nfRi+fBpQN3Jl0j2Hj0oKxFfrOkpzNxxMHrnr3euq/QDURMZTevrPSD7zNneD2NFa/iK8eRvc/IG7\nD8OzI6+U2UszeXVpJtlFFSR3iuay43tx1YQ+jOqV1Oxvt8UV1cxeksnzi7azPa+MbomxXHdSX647\nsTfdFv4Svnjcfam44umgvzjsr6zhV2+v5aXFuxjWPZGHrxnLqF5Hd5Pczrwy/vzBRt5YnkV8TBTf\nOn0AN08aQFLc4U1BqsryXYXMWZHN26t2s29/JYmxUVwwugeXHt+LUwZ2JWrta/DGrdBloKslpfSB\nvWthwW9cTTsuBU77vrtL3O9vz+dTdhdXsCVn/4FayJbc/XRPiuMv1447qt+tLSSLacAUVb3Ze/0N\n4CRVvcOvTE9gHpAKJADnqupSL1mswdVMioGfq+onAa5xC3ALQN++fcfv2LEjJL+LOQqVJbB2jqtF\nbPsYUPetcMw1MOqK4L4ZVu6HHQtd4tgy3w0RBIjv6pqqBnnJI6WvK7vpXXfNTfOgugw6dXFz9oy8\nDAacAVHBfRsE3Lf6Dx9w/3ET0uGMH7spHY7kHOCahBY9Cov+5mIaOx0mz3AxH4Wyqhq+2JrPx5ty\n+XhDDmX7djEoIptNvt7spQs9k+MY3y/1wGNEzySig/wW7eItgj8MhhNvpvSsXzL3q928sjSTL7fl\nEyEwaUg6V0/ow7kjuxEb1Yyk1ACfT/loUy7PfbadLzZm8kjMY5wfsZicUd+m25V/CDoRfr41jx+9\nspLswnJuOWMQd583pEXi3bS3hIff28g7q/eQ3Cma7505iBtO7Ud8TBQb95YwZ0U2c1ZmszO/jJio\nCM4Z3o1Lj+/F5GHdiIuud/1tn7iO75h492VozRuu1nvK7XDKba1293d7SRY/9GL4o1ez+AcwGogG\nOqtqnoiMB94ARqlqcUPXs5pFG1BbA1s+dAli/X+hpty1/Y+5xj2aO36/ZA9sXeDVPBbA/j1ue0o/\n2L/XNdUkdIMRF7kaRL/Tmz1ShF2L4f2ZbjqM1P5w9i9csvNqNnn7K9mwt4TYqAj6dU2ga4LXhlxd\n7jqMP3nYjcAZeakblpo+7Igu7/Mpa3cX88mmfXy8MZelOwqoqvURGxXBSQO7csaQNE4fkkZ1jbJ0\nRz5LdhSwbEcB2UWu2apTdCRj+yQzvl8qE/p1YVzfFFLiG054qkrhs1cTkb2U06v+RkmVMiAtgWnj\nM7jyhAx6JB99m/0RKd1HxfNXEbt3Ob/TG3ii8nzGZiRzw6n9+dqYng1+8FdU1/KHdzfwzGfb6Ncl\nnj9ePZbx/bq0eHirs4p4aN4GFmzIJa1zLGmdY1i/p4QIgdMGp3HJ2F5cMLpHwJrHIfaudSO7Kgrd\njLWnfh/iWz7exrSFZBFMM9QaXELZ5b3eCpysqjn1zrUA+JGqNpgNLFmE0Z7Vrtlm9atQmuva30dd\n4RJEn4mhGeWk6ppitsx3o5qSersP5L4nN68ZpoFr1WycR828+4jLW8ee+KE8n3ATswuGkLO/6pCi\nKbHw7YSFXF/1Eqk1+9idfhr7Jv6Y9KEn0y0xloiGhkb6ySmucMlhUy6fbtpHXqm7xvAeiZwxNJ0z\nhqQzoX/q4d9U/WQXlrN0RwFLdxSwbGcBa7KLqfW5/+uDu3VmfN9Uxvd3tY+BaQnsKa7gtWVZzF6a\nycj8D3g05hGe6P8Xxk++hAn9UpvdzHRE6g2N3T9wKq8ty+SfC7ezJbeUtM4xTJ/Yl6+f1O+Q5LUq\ns5C7X17BltxSvnlKP2ZMHU58TGjvO168PZ9HPthEeVUtF43pydfG9CI98QgnEKwocveDNHDjYKi1\nhWQRhWtGOgfIwnVwX6eqa/zKvAO8rKrPicgI4AOgN5AG5KtqrYgMBD4BjlPV/IauZ8kiTL54yo3A\niYh0/Q9jroEh5x95c81RUlX2FFdQU6vEx0SSEBtFbFREsz7ccksqWb+nmPW7S1i3u5h1e0rYnFNC\nTW0tl0Ys5EfRs8iQfWyKP4E1o35I16EnU1NTS8SaVzlu06N0qcxidcRwflN5FQtrD07zEBcdQb8u\nCfTrGk//NO9n1wT6pMazI7+Ujzfm8smmfazfUwJAWucYTh+cxhlD0zl9cBrdmjESp6yqhpW7ili6\nI/9AEimuqAFcJ2lJRTU+hZMGdOHa49O47P0zkTFXwcV/OeprHpVdX7qb7QIMjVVVPt28j+c+286H\nG3KIFGHK6B5885T+fLp5H4/O30y3xFh+P20Mk4akt27c7VjYk4UXxIXAn3HDYp9R1V+LyAPAElWd\n442AehroDCjwY1WdJyJXAg8A1YAPuE9V32rsWpYsWlltjUsSi592SeKyx0Nefa6q8bEldz9rs4tZ\nu7v4wM+i8upDykUIJMREER8beeBnfEwUCTGRxMd6P2OiSPC2d4qOZHdROev3lLBudwn79lceOFf3\npFhG9ExieI8kRvRMZETPJAakRBG9/Dn4+A9u+O2IiyF/G+xdDd2Pg3N+AUPOp8an7C6qYHteKdvz\nytixz/uZV8qO/DKqag5d9yAmMoIJ/VOZNCSdM4amMaJHUlA1kaPh8ylbcvezdEcBy3cW0j0plivH\nZ9Cva4Ir8OrNsPl9+NGmFhvP36QjGBq7I6+UFxbt4OUluyjxkt4VJ/TmvotHkdypleLtINpEsmhN\nlixaUXkhzL7J9U+ceiece3+LN/0UlVezzi8hrM0uZnPOfqpq3QdsbFQEw3smMbKn+xDvFB1JWVUt\npVU1lFXW+1lVS2llzYH95d7r0qraA00zMVERDO3emRE9khjeM4kRPRIZ3jOp8SGSFcWw8K+uAzux\nu+uT8OvPaIzP52pE2/NK2ZVfRrfEOE4a2CXkzSZB2/COu/ntuldg6Pmhv95RDo0tq6rhv6t20y0p\njjOHWm3iaFiyMKGRv9U1E+RvhYv+BCd885DdPp9Sq0qtT/HV/fRx2Db/5zU+Zfu+0kNqC5kF5QfO\nmdY5hpG9kg8khlG9kujfNSHosfINUVWqan2UVdaSGBd19OerrXZ3ADd3ptO2pKYKHhoMQ6fCFU+G\n9lof/NLNhnuEQ2NNywg2WbSRrzGmXdj+Gbx8PaDUXv8GX+pI3n79K+at3Ut+adWBb+lHSwQGpiUw\nrm8q153Ul5E9kxjZK4luiaEZgSMixEZFNn9IZWs107SmqBgYcYlb46K6PHQf4LtXwSd/dNNXXPLX\nlh+cYFqMJQsTnOX/Qt/6ARWd+/BUxm/5138qyS35nE7RkZw9ohsDuiYQESFEihAZgd9zIcKb0iEy\nQg5sP2R/hNAntRPDeiS2nWYYA8dNc6PcNs1zI81amqqbFr1TqptgzxJFm2b/M02jtLaGnDd+Rvev\nnmSxjOHmnDuozFfOHp7KRWN6cfbwbnSKsf/kHVL/Se6+la9mhyZZbHzX3bA59feB57sybYoli3au\n7iaksqoa+nSJp2+XePp1SaBvl3iS44+ueURV+SqriHeXb+Hk5T9hkm8xL9aezyeD7+GXY/twzoju\ndI61P50OLyISRl0Oy/7pOvObmuL8SNRWw7yfQ9fBbuoT0+bZ//h2zOdT7nllJf9dtZuuCTEHbt6q\nk9wpmr5eAunbNf7g8y7x9EyOO6RDV9XdKfzfVbv571e7qc7byT9i/8gw2cWK0T/j4ot+yNebuhvV\ndDyjr4Qvn4QNc91yoS1l6XOQtwmu/U/H7PPpgCxZtGO/+996/rtqNz+7cDi3nDGI/ZU17MovY2d+\nGTvzvJ/5ZazdXcy8tXuorj3YAR0VIWSkdqKPlziWbC9g675SIiOEb2Tk8JOaB4jTKuSqVzh+yLlh\n/C1NWGWc6KZKX/1qyyWL8kJY8FvXzDVsasuc04ScJYt26oVF23ny461885R+fGfSQAA6x0YxomcS\nI3oe3lxQ61N2F5WzM7+MXfll7PCSya78MtZmFzOsRyI3TxrIxZELSXznx5DYA66b5RYFMseuiAi3\nFOiiR92Kby1x4+Unf3TnOv9XbXtFPnMISxbt0Ptr93LfnDWcO6Ib9108KqipLSIjhIzUeDJS4yHQ\njbGqbmrujx6EvqfANf8KzQJEpv0ZfSV89hd3h/WEm5p3roLtbs2QsdOh1/EtEp5pHR3oLqJjw6rM\nQu78z3JG907mkenjGl6z90hUl8Psb7lEMfY6+OablijMQT3GQNchrimqud6f6W5gPOcXzT+XaVVW\nswg3VfetzVfj1j3uNgKS+wa8G3hXfhnfem4xXTvH8I8bTjz6exKqK9zaEHu+co+tC9ySnufe75bU\ntKYB40/E1S4++h0U74aknkd3nl1fupv8zvyJm//JtCuWLMKtOAver7fecUxnlzS6jXSP7iMpThzK\njf9cT3Wt8tItJwY/DXLpvoNJoe6xbyOotzRjdAL0GA3X/tutJGdMIKOvdDXPtW+4pWCPlKqb+6lz\nd7dmg2l3LFmEW463+tt1r7gbk/augZy1bvu6t9wYdyAJ+I+mENtrFMnLFrg1lruNdOswx8SDz+fm\na9pbLzGU7D54raTe0H20Wz2ux3HukTqgY81pZEIjfaj7e/lq9tElizWvQeZiuORvzV+i1oSFJYtw\nq0sWGRPcSJM+Ew/uU8VXvIfHX3mLgm0r+ObAUrrVbIcl/3CrwgEgbonO0n1QXeptinRJZMCZrtbQ\n4zg3dfYRLHJvzGFGT3O14ILtbtXAYFVXuL6K7qPh+OtCFJwJNUsW4Za73lXNAw1JFOEPi4p4fHNv\nfjzlbPpOHuy2+2rd+gk5a9yyjPs2uHP0OM79h0wfDtGttPylOXaMvsIli9WvwqR7gj/uiyegcCd8\n4w2b/6kds2QRbjnr3Id7AC9+sYPHF2zhupP6cuuZfuNdIyIhbbB7hGLOHmMCSekLfU6C1a8FnyxK\n97n7KoZcAIPOCm18JqSssTqcfD7I3eA6s+uZvz6HX7yxmrOGpfPAJcHdS2FMyI2+0q0ImLM+uPIL\nfgtVpXD+L0Mblwk5SxbhVLTL9TPUq1msziri9n8vY2SvJP523QnNXuTHmBYz8jKQiODuucjdAEue\ndTfypQ8LfWwmpOxTKJxyvW9nfjWLzIIybnpuManxMTxzw4kk2Oyupi1J7O7mdFo92w2Hbcx790JM\nAkz+aevEZkLKkkU41Y2E8moWReXV3PTsYiqqa3n2phPplmSd1KYNOm6aG6a9e0XDZbYugI3/c30b\nNhtAh2DJIpxy10NiT+iUQmVNLd99YQnb80p58hvjGdo9MdzRGRPYiIshItrdcxGIrxbe/bmbieCk\n77VubCZkLFmEkzcSSlWZ8epXfL41n99PG8Opg+ybmGnDOqXC4HPd1B0+3+H7V/zb3Rx63kwbwt2B\nhDRZiMgUEdkgIptFZEaA/X1FZL6ILBeRVSJyYYD9+0XkR6GMMyx8PjftRrcRPPzeRl5fnsWPzh/K\n5eMywh2ZMU0bfaWbqmbX54dur9wPH/7KrYMx6orwxGZCImTJQkQigUeBqcBIYLqIjKxX7OfALFUd\nB1wLPFZv/8PAO6GKMawKd0B1Gaure/LXDzdzzYQ+3H7W4HBHZUxwhk2FqE6Hj4pa+Ajs3wMX/MYm\npOxgQlmzmAhsVtWtqloFvATUv4NMcdMeASQD2XU7ROQyYBuwJoQxho83EurBJcLYPin88rLRdi+F\naT9iO8OwKbDmDaitcduKs+GzR9y63f7T1pgOIZTJojewy+91prfN30zgehHJBOYCdwKISGfgJ8D9\njV1ARG4RkSUisiQ3N7el4m4V1XvWArBNMvjb9HHERFn3kWlnRl8JZftg20fu9Ye/crMZnzsznFGZ\nEAn3J9R04DlVzQAuBF4QkQhcEvmTqu5v7GBVfUpVJ6jqhPT09NBH24JWr/iCbO3CA9ecSp8u8eEO\nx5gjN/g8iE1yTVG7V7qO7ZO+d2STDJp2I5R3fGUBffxeZ3jb/H0bmAKgqotEJA5IA04CponI74EU\nwCciFar6txDG22reXJHFoLwNVKQO5ZwR3cMdjjFHJzoOhl/kptLP3+ZGSR3JBIOmXQllzWIxMERE\nBohIDK4De069MjuBcwBEZAT0vh8iAAAgAElEQVQQB+Sq6iRV7a+q/YE/A7/pKIlic85+/u+1lQyN\nyKbfiPHhDseY5jnuSqgshp0L4ayfuTVZTIcUspqFqtaIyB3Au0Ak8IyqrhGRB4AlqjoHuAd4WkTu\nxnV236ja1BwC7Vd5VS23vbiUQVH7iPFVBZxA0Jh2ZcCZEJ/mahXjbwx3NCaEQjrxkKrOxXVc+2+7\n1+/5WuC0Js4xMyTBhcEv3lzNppz9vH1eFHyMJQvT/kVGwzdec30XkdHhjsaEULg7uI8Zs5bsYvbS\nTO48ewijorwRwjYTp+kIeo6FLgPCHYUJMUsWrWDd7mJ+8cZqTh3UlbvOGeLusUjuA7E2/5Mxpn2w\nZBFi+ytruP3FZSR1iuYv144jMkLcwjENrI5njDFtkSWLEFJVfvraV2zPK+Wv08eRnhjrZuTctxG6\nWbIwxrQflixC6F9f7OStldncc/4wTh7Y1W3M3wa1lZBundvGmPbDkkWIrM4q4pdvrWXysHRuPXPQ\nwR253oJHVrMwxrQjlixCoKi8mtteXEbXzjE8fPXxRET4TRBYt9B9mo2EMsa0H7bAcwtTVX48eyXZ\nheW8/N2T6ZIQc2iB3HWQ0tfN2mmMMe2E1Sxa2DOfbefdNXuZMXU44/t1ObxAznroVn9ZD2OMadss\nWbSgZTsL+O3cdZw3sjvfPj3ATUq1NZC3yYbNGmPaHUsWLaSgtIo7XlxGz5Q4Hpo2NvBCRvlbodbm\nhDLGtD/WZ9ECfD7lh7NWsG9/FbNvPYXk+AbmyKkbCWU1C2NMO2M1ixbwxMdbmL8hl19cNIIxGY1M\n0ZyzHhBIG9pqsRljTEuwZNFMJRXVPDxvI187rifXn9yv8cK569wqYjG2Mp4xpn2xZFFeAAv/CnvX\nHNXhO/PLqPEpF43pGbifwl/OeuuvMMa0S5YsAOb9AtbWX8QvONmFFQD0SunUeMHaasjbbP0Vxph2\nyZJFp1ToNQ62Ljiqw7MKygDondpEssjbAr5qq1kYY9olSxYAg86CzMVQUXzEh2YXVRAbFUHX+ndq\n12cjoYwx7ZglC4CBk0FrYcfCIz40q6Cc3imdguuvkAgbCWWMaZcsWQBkTISoTkfVFJVVWN50fwV4\nI6EGQHTckcdnjDFhZskC3Ad4v1Ng6/wjPjSr0NUsmmQjoYwx7VhIk4WITBGRDSKyWURmBNjfV0Tm\ni8hyEVklIhd62yeKyArvsVJELg9lnIBrispdD8W7gz6ksqaW3JLKpmsWNVWQv8X6K4wx7VbIkoWI\nRAKPAlOBkcB0Eak/3erPgVmqOg64FnjM274amKCqxwNTgCdFJLRTkwyc7H5u+yjoQ3Z7w2abHgm1\nGXw1VrMwxrRboaxZTAQ2q+pWVa0CXgIurVdGgSTveTKQDaCqZapa422P88qFVvfjIL7rEfVbZBeW\nA9ArpYl+CBsJZYxp50KZLHoDu/xeZ3rb/M0ErheRTGAucGfdDhE5SUTWAF8B3/NLHviVuUVElojI\nktzc3OZFGxEBA850yUKDy02ZXrLISGli+o6c9SCRkDakeTEaY0yYhLuDezrwnKpmABcCL4hIBICq\nfqGqo4ATgZ+KyGFf31X1KVWdoKoT0tPTmx/NwMlQshv2bQyqeHZhOSLQIzmImkWXgRAV2+wQjTEm\nHEKZLLKAPn6vM7xt/r4NzAJQ1UW4Jqc0/wKqug7YD4wOWaR1Bk52P4NsisoqKKdbYiwxUU28jTnr\noJs1QRlj2q9QJovFwBARGSAiMbgO7PoTMO0EzgEQkRG4ZJHrHRPlbe8HDAe2hzBWJ7WfuxdiS3BD\naLOLgrjHorrCLXqUbp3bxpj2K2TJwutjuAN4F1iHG/W0RkQeEJFLvGL3AN8RkZXAf4AbVVWB04GV\nIrICeB24TVX3hSrWQwycDNs/dRP/NaHu7u1G5W0C9VnNwhjTrgU1HFVEXgP+Abyjqr5gT66qc3Ed\n1/7b7vV7vhY4LcBxLwAvBHudFjVwMix9FrKWQd+TGizm8ynZRRVcMKpH4+fLWe9+Ws3CGNOOBVuz\neAy4DtgkIg+KyLAQxhReA84ApMl+i32llVTV+Jq+xyJ3HUREQdfBLRaiMca0tqCShaq+r6pfB07A\n9R28LyILReQmEWlgwel2Kr4L9BzbZLI4sI5FchPJImc9dBkEUU3MSmuMMW1Y0H0WItIVuBG4GVgO\n/AWXPN4LSWThNOgsyPwSKvc3WCSrwN1jEVTNwvorjDHtXFDJQkReBz4B4oGLVfUSVX1ZVe8EOocy\nwLAYONlNz7HjswaLHLx7u5FkUV0O+dusv8IY0+4FO9/SI6oacDypqk5owXjahj4nQ1Sca4oaekHA\nIlmF5STGRpHcqZFWuH0bAbWahTGm3Qu2GWqkiKTUvRCRVBG5LUQxhV90HPQ9udF+i6DWsbCRUMaY\nDiLYZPEdVS2se6GqBcB3QhNSGzFwMuSshZK9AXdnFZQHORIqGroOavHwjDGmNQWbLCLFb91Qb/rx\njj28Z+Bk97OBKctdzaKJOaFy1rshs5Eda8CYMebYE2yy+B/wsoicIyLn4O62/l/owmoDeoyBTqkB\nm6L2V9ZQVF5N76Zmm7WRUMaYDiLYDu6fAN8FbvVevwf8PSQRtRURkYdOWX6wYhXcOhZVZVCwA8Ze\nF+JAjTEm9IJKFt4UH497j2PHwMmw9g3YtwnShx7YnFW3jkVjfRb7NmAjoYwxHUWw91kMEZHZIrJW\nRLbWPUIdXNgNnOx+1muKOnBDXmPNUDYSyhjTgQTbZ/EsrlZRA5wFPA/8K1RBtRldBkBKv8OSRXZh\nOVERQnpiI4sZ5a6DyBi36JExxrRzwSaLTqr6ASCqukNVZwJfC11YbcjAybD9E6g9uKprVmE5PVPi\niIyQBg9zI6GGQGSw3ULGGNN2BZssKr3lTjeJyB0icjkdcZqPQAZOhspiyF5+YFN2YXnTEwjaSChj\nTAcSbLK4Czcv1PeB8cD1wA2hCqpNGXAm9acsb/KGvMr9ULjT+iuMMR1Gk8nCuwHvGlXdr6qZqnqT\nql6pqp+3Qnzhl9AVeo45kCxqan3sKa5ofIW8fRvcT6tZGGM6iCaTharW4pY5PXYNnAy7voDK/ewp\nrsCnNJ4sbCSUMaaDCbYZarmIzBGRb4jIFXWPkEbWlgycDL5q2Lno4KJHjSWL3HUQGetGUxljTAcQ\n7FCdOCAPONtvmwKvtXhEbVHfU9yH/9YFZHVztYVG+yxy1rub+CIiWylAY4wJrWDv4L4p1IG0adGd\noO9JsHUB2THurWh0NFTuepdgjDGmgwj2Du5nReSZ+o8gjpsiIhtEZLOIzAiwv6+IzBeR5SKySkQu\n9LafJyJLReQr7+fZh5+9lQ2cDHtXU5iTRdeEGDrFNFBrqCyBol3WuW2M6VCC7bN4G/iv9/gASAIa\nXqCaA6OoHgWmAiOB6SIysl6xnwOzVHUccC3wmLd9H2751uNwQ3RfCDLO0Bk4GYAuOYua6K/wRkJZ\n57YxpgMJthnqVf/XIvIf4NMmDpsIbFbVrd4xLwGXAmv9T41LPADJQLZ3veV+ZdYAnUQkVlUrg4k3\nJHoeD3EpDCj+kt59z2u4XM4699NqFsaYDiTYmkV9Q4BuTZTpDezye53pbfM3E7heRDKBucCdAc5z\nJbAsrIkCICISHXAGY6tW0Cu5kanJc9dDVCdI6d9qoRljTKgF22dRIiLFdQ/gLdwaF801HXhOVTOA\nC4EXvGlF6q47Cvgdbi2NQHHdIiJLRGRJbm5uC4TTuLKMSfSUPEbENnKtnHXeSKijzcPGGNP2BPWJ\npqqJqprk9xhav2kqgCygj9/rDG+bv28Ds7xrLMIN0U0DEJEM4HXgm6q6pYG4nlLVCao6IT09PZhf\npVmyUk8CYGTF0oYL5a63/gpjTIcTbM3ichFJ9nudIiKXNXHYYmCIiAwQkRhcB/acemV2Aud45xyB\nSxa5IpKC60yfoaqfBferhN42XzcyNY2M/C8DF6goguIs668wxnQ4wbaV3KeqRXUvVLUQuK+xA1S1\nBrgDeBdYhxv1tEZEHhCRS7xi9wDfEZGVuHW9b1RV9Y4bDNwrIiu8R1N9JCGXXVTBp7WjSdyzEHy1\nhxewkVDGmA4q2Du4AyWVJo9V1bm4jmv/bff6PV8LnBbguF8BvwoytlaTVVBOfsQYrq1cANkrIGP8\noQVsJJQxpoMKtmaxREQeFpFB3uNhoJGG+44pu6ic7Ylegtj64eEFctdDdDwk923dwIwxJsSCTRZ3\nAlXAy8BLQAVwe6iCaquyCspJ6NITehwHWz86vEDOOkgfZiOhjDEdTrA35ZUCh03XcazJKqxgRM8k\nSJwMXzwJVaUQk3CwQO56GHhW2OIzxphQCXY01HveCKW616ki8m7owmp7Kqpr2be/0q1jMXAy1FbB\nzkUHC5QXQslu668wxnRIwbaXpHkjoABQ1QKavoO7Q9ld5LeORd9TIDLmkKVWybUFj4wxHVewycIn\nIgd6bUWkP25ep2NGVkE54K1jEZMAfU46NFnYSChjTAcWbLL4P+BTEXlBRP4FfAT8NHRhtT3ZhV6y\nqJtxduCZsOcr2O9N/ZG7HmI6Q3KfBs5gjDHtV7DTffwPmABswN08dw9QHsK42pzMwnJEoEfdJIJ1\nHdnbvFFRdSOhRMIToDHGhFCwHdw349axuAf4EW59iZmhC6vtyS4sp3tiHNGR3lvWaxzEJh9sirI5\noYwxHViwzVB3AScCO1T1LGAcUNj4IR1LVkH5oetuR0TCgEkuWZTlw/691l9hjOmwgk0WFapaAeAt\nQrQeGBa6sNqe7KLyw1fIGzjZLaG6wZvRxGoWxpgOKthkkendZ/EG8J6IvAnsCF1YbYvPp+wurDjY\nuV2nrt/i88fdT6tZGGM6qGDv4L7cezpTRObjlkD9X8iiamP27a+kqtZH75R6K+R1HQRJGbB3NcQm\nQVL9hQCNMaZjOOJJjFT1I1Wdo6pVoQioLcos9LvHwp+Ia4oCGwlljOnQbMa7IBy4IS8l/vCdAye7\nn+nWBGWM6bgsWQSh7oa8XvWbocAli6g4yJjQqjEZY0xrCnbxo2NaVmE5SXFRJMZFH76zczrctRIS\nQr8GuDHGhIsliyBkFwYYNusvsUfrBWOMMWFgzVBByCwoJ6N+57YxxhxDLFkEocmahTHGdHCWLJpQ\nUlFNcUXN4TfkGWPMMcSSRROyC/0WPTLGmGNUSJOFiEwRkQ0isllEDlvDW0T6ish8EVkuIqtE5EJv\ne1dv+34R+VsoY2xKVmEZEOCGPGOMOYaELFmISCTwKDAVGAlMF5GR9Yr9HJilquOAa4HHvO0VwC9w\n06GHVZZXs7BmKGPMsSyUNYuJwGZV3epNDfIScGm9Mgokec+TgWwAVS1V1U9xSSOssgrKiY4U0jvH\nhjsUY4wJm1Ami97ALr/Xmd42fzOB60UkE5gL3HkkFxCRW0RkiYgsyc3NbU6sDcouLKdnciciImze\nJ2PMsSvcHdzTgedUNQO4EHhBRIKOSVWfUtUJqjohPT00d1BnFZZbE5Qx5pgXymSRBfTxe53hbfP3\nbWAWgKouAuKAtBDGdMTsHgtjjAltslgMDBGRASISg+vAnlOvzE7gHAARGYFLFqFpTzoK1bU+9hZX\n2EgoY8wxL2RzQ6lqjYjcAbwLRALPqOoaEXkAWKKqc4B7gKdF5G5cZ/eNqqoAIrId1/kdIyKXAeer\n6tpQxRvInqIKfMrhix4ZY8wxJqQTCarqXFzHtf+2e/2erwVOa+DY/qGMLRhZhY2sY2GMMceQcHdw\nt2mNrmNhjDHHEEsWjahbIc86uI0xxzpLFo3ILionrXMMcdGR4Q7FGGPCypJFIzIL7B4LY4wBSxaN\nsnssjDHGsWTRAFW1u7eNMcZjyaIBBWXVVFT7rGZhjDFYsmhQ3Ugou3vbGGMsWTTo4A15liyMMcaS\nRQMsWRhjzEGWLBqQXVhOp+hIUuKjwx2KMcaEnSWLBmQVlNM7tRMituiRMcZYsmhAdpHdY2GMMXUs\nWTQgy+7eNsaYAyxZBFBRXUteaZWtY2GMMR5LFgEcGAll91gYYwxgySKgbFv0yBhjDmHJIoCD61hY\nM5QxxoAli4CyC8uJEOiRZMnCGGPAkkVAmYXl9EiKIyrS3h5jjAFLFgFlF5Zb57YxxvgJabIQkSki\nskFENovIjAD7+4rIfBFZLiKrRORCv30/9Y7bICIXhDLO+rJs0SNjjDlEyJKFiEQCjwJTgZHAdBEZ\nWa/Yz4FZqjoOuBZ4zDt2pPd6FDAFeMw7X8jV+pTdhRV2Q54xxvgJZc1iIrBZVbeqahXwEnBpvTIK\nJHnPk4Fs7/mlwEuqWqmq24DN3vlCLrekkhqfWs3CGGP8hDJZ9AZ2+b3O9Lb5mwlcLyKZwFzgziM4\nFhG5RUSWiMiS3NzcFgk6q7DMBWB9FsYYc0C4O7inA8+pagZwIfCCiAQdk6o+paoTVHVCenp6iwSU\nVVgB2DoWxhjjLyqE584C+vi9zvC2+fs2rk8CVV0kInFAWpDHhsTBG/IsWRhjTJ1Q1iwWA0NEZICI\nxOA6rOfUK7MTOAdAREYAcUCuV+5aEYkVkQHAEODLEMZ6QHZhOcmdoukcG8o8aowx7UvIPhFVtUZE\n7gDeBSKBZ1R1jYg8ACxR1TnAPcDTInI3rrP7RlVVYI2IzALWAjXA7apaG6pY/WUV2tTkxhhTX0i/\nPqvqXFzHtf+2e/2erwVOa+DYXwO/DmV8gWQXlpORahMIGhNu1dXVZGZmUlFREe5QOoS4uDgyMjKI\njj66paKtraWerIJyTh7YNdxhGHPMy8zMJDExkf79+9vyxs2kquTl5ZGZmcmAAQOO6hzhHg3VphRX\nVFNSWWOzzRrTBlRUVNC1a1dLFC1AROjatWuzammWLPzUjYSydSyMaRssUbSc5r6Xliz81C16ZDUL\nY4w5lCULP7acqjGmTmFhIY899tgRH3fhhRdSWFjYaJl7772X999//2hDCwtLFn6yCsuJiYwgLSE2\n3KEYY8KsoWRRU1PT6HFz584lJSWl0TIPPPAA5557brPia202GspPVkE5vVLiiIiwdlJj2pL731rD\n2uziFj3nyF5J3HfxqAb3z5gxgy1btnD88ccTHR1NXFwcqamprF+/no0bN3LZZZexa9cuKioquOuu\nu7jlllsA6N+/P0uWLGH//v1MnTqV008/nYULF9K7d2/efPNNOnXqxI033shFF13EtGnT6N+/Pzfc\ncANvvfUW1dXVvPLKKwwfPpzc3Fyuu+46srOzOeWUU3jvvfdYunQpaWlpLfo+BMtqFn6ybR0LY4zn\nwQcfZNCgQaxYsYI//OEPLFu2jL/85S9s3LgRgGeeeYalS5eyZMkSHnnkEfLy8g47x6ZNm7j99ttZ\ns2YNKSkpvPrqqwGvlZaWxrJly7j11lt56KGHALj//vs5++yzWbNmDdOmTWPnzp2h+2WDYDULP1mF\n5ZwxpGUmJDTGtJzGagCtZeLEiYfco/DII4/w+uuvA7Br1y42bdpE166H3qM1YMAAjj/+eADGjx/P\n9u3bA577iiuuOFDmtddeA+DTTz89cP4pU6aQmpraor/PkbJk4amq8ZFTUmk1C2NMQAkJCQeeL1iw\ngPfff59FixYRHx/P5MmTA97DEBt7sP8zMjKS8vLygOeuKxcZGdlkn0i4WDOUZ09RBao2EsoY4yQm\nJlJSUhJwX1FREampqcTHx7N+/Xo+//zzFr/+aaedxqxZswCYN28eBQUFLX6NI2E1C8+BYbNWszDG\nAF27duW0005j9OjRdOrUie7dux/YN2XKFJ544glGjBjBsGHDOPnkk1v8+vfddx/Tp0/nhRde4JRT\nTqFHjx4kJia2+HWCJW6S1/ZvwoQJumTJkqM+fvbSTH70ykoW/Ggy/dMSmj7AGBNS69atY8SIEeEO\nI2wqKyuJjIwkKiqKRYsWceutt7JixYpmnTPQeyoiS1V1QlPHWs3CU3f3do9ku3vbGBN+O3fu5Oqr\nr8bn8xETE8PTTz8d1ngsWXiyCspJT4wlLjoy3KEYYwxDhgxh+fLl4Q7jAOvg9mQX2T0WxhjTEEsW\nnqyCcjIsWRhjTECWLHALg2QVlttss8YY0wBLFkBeaRWVNT4bNmuMMQ2wZIH/OhaWLIwxR6dz584A\nZGdnM23atIBlJk+eTFND/P/85z9TVlZ24HUwU563BksW+K2QZ3dvG2OaqVevXsyePfuoj6+fLIKZ\n8rw12NBZDt69nWHLqRrTNr0zA/Z81bLn7HEcTH2wwd0zZsygT58+3H777QDMnDmTqKgo5s+fT0FB\nAdXV1fzqV7/i0ksvPeS47du3c9FFF7F69WrKy8u56aabWLlyJcOHDz9kbqhbb72VxYsXU15ezrRp\n07j//vt55JFHyM7O5qyzziItLY358+cfmPI8LS2Nhx9+mGeeeQaAm2++mR/84Ads3769wanQW1JI\naxYiMkVENojIZhGZEWD/n0RkhffYKCKFfvt+JyKrvcc1oYwzq7CchJhIkjpZ7jTGONdcc82BuZkA\nZs2axQ033MDrr7/OsmXLmD9/Pvfccw+NzYLx+OOPEx8fz7p167j//vtZunTpgX2//vWvWbJkCatW\nreKjjz5i1apVfP/736dXr17Mnz+f+fPnH3KupUuX8uyzz/LFF1/w+eef8/TTTx+4DyPYqdCbI2Sf\njiISCTwKnAdkAotFZI6qrq0ro6p3+5W/ExjnPf8acAJwPBALLBCRd1S1ZVc/8WQXltM7tZMtDm9M\nW9VIDSBUxo0bR05ODtnZ2eTm5pKamkqPHj24++67+fjjj4mIiCArK4u9e/fSo0ePgOf4+OOP+f73\nvw/AmDFjGDNmzIF9s2bN4qmnnqKmpobdu3ezdu3aQ/bX9+mnn3L55ZcfmP32iiuu4JNPPuGSSy4J\neir05gjlV+mJwGZV3QogIi8BlwJrGyg/HbjPez4S+FhVa4AaEVkFTAFmNXBss2TZokfGmACuuuoq\nZs+ezZ49e7jmmmt48cUXyc3NZenSpURHR9O/f/+AU5M3Zdu2bTz00EMsXryY1NRUbrzxxqM6T51g\np0JvjlA2Q/UGdvm9zvS2HUZE+gEDgA+9TSuBKSISLyJpwFlAnwDH3SIiS0RkSW5u7lEHml1YYcNm\njTGHueaaa3jppZeYPXs2V111FUVFRXTr1o3o6Gjmz5/Pjh07Gj3+jDPO4N///jcAq1evZtWqVQAU\nFxeTkJBAcnIye/fu5Z133jlwTENTo0+aNIk33niDsrIySktLef3115k0aVIL/raNayuN9NcCs1W1\nFkBV54nIicBCIBdYBNTWP0hVnwKeAjfr7NFcuKyqhvzSKqtZGGMOM2rUKEpKSujduzc9e/bk61//\nOhdffDHHHXccEyZMYPjw4Y0ef+utt3LTTTcxYsQIRowYwfjx4wEYO3Ys48aNY/jw4fTp04fTTjvt\nwDG33HILU6ZMOdB3UeeEE07gxhtvZOLEiYDr4B43blxImpwCCdkU5SJyCjBTVS/wXv8UQFV/G6Ds\ncuB2VV3YwLn+DfxLVec2dL2jnaI8v7SK++as4eoJGUyyJVWNaTOO9SnKQ6GtTlG+GBgiIgOALFzt\n4br6hURkOJCKqz3UbYsEUlQ1T0TGAGOAeaEIsktCDH+dPi4UpzbGmA4jZMlCVWtE5A7gXSASeEZV\n14jIA8ASVZ3jFb0WeEkPreJEA594o5OKgeu9zm5jjDFhENI+C6/ZaG69bffWez0zwHEVuBFRxphj\nmKrakPYW0twuB5vuwxjTJsXFxZGXl9fsDznjEkVeXh5xcUc/s3ZbGQ1ljDGHyMjIIDMzk+YMizcH\nxcXFkZGRcdTHW7IwxrRJ0dHRDBgwINxhGI81QxljjGmSJQtjjDFNsmRhjDGmSSG7g7u1iUgu0PhE\nLY1LA/a1UDihYPE1j8XXPBZf87Tl+PqpapPTV3SYZNFcIrIkmFvew8Xiax6Lr3ksvuZp6/EFw5qh\njDHGNMmShTHGmCZZsjjoqXAH0ASLr3ksvuax+JqnrcfXJOuzMMYY0ySrWRhjjGmSJQtjjDFNOqaS\nhYhMEZENIrJZRGYE2B8rIi97+78Qkf6tGFsfEZkvImtFZI2I3BWgzGQRKRKRFd7j3kDnCnGc20Xk\nK+/6hy1NKM4j3nu4SkROaMXYhvm9NytEpFhEflCvTKu+hyLyjIjkiMhqv21dROQ9Ednk/Uxt4Ngb\nvDKbROSGVozvDyKy3vv3e11EUho4ttG/hRDGN1NEsvz+DS9s4NhG/7+HML6X/WLbLiIrGjg25O9f\ni1LVY+KBW4BpCzAQiAFWAiPrlbkNeMJ7fi3wcivG1xM4wXueCGwMEN9k4O0wv4/bgbRG9l8IvAMI\ncDLwRRj/vffgbjgK23sInAGcAKz22/Z7YIb3fAbwuwDHdQG2ej9TveeprRTf+UCU9/x3geIL5m8h\nhPHNBH4UxL9/o//fQxVfvf1/BO4N1/vXko9jqWYxEdisqltVtQp4Cbi0XplLgX96z2cD50grrbyi\nqrtVdZn3vARYB/RujWu3sEuB59X5HEgRkZ5hiOMcYIuqNueu/mZT1Y+B/Hqb/f/O/glcFuDQC4D3\nVDVfVQuA94AprRGfqs7TgytTfg4c/bzWzdTA+xeMYP6/N1tj8XmfHVcD/2np64bDsZQsegO7/F5n\ncviH8YEy3n+WIqBrq0Tnx2v+Ggd8EWD3KSKyUkTeEZFRrRqYo8A8EVkqIrcE2B/M+9warqXh/6Th\nfg+7q+pu7/keoHuAMm3lffwWrqYYSFN/C6F0h9dM9kwDzXht4f2bBOxV1U0N7A/n+3fEjqVk0S6I\nSGfgVeAHqlpcb/cyXLPKWOCvwButHR9wuqqeAEwFbheRM8IQQ6NEJAa4BHglwO628B4eoK49ok2O\nXxeR/wNqgBcbKBKuv4XHgUHA8cBuXFNPWzSdxmsVbf7/kr9jKVlkAX38Xmd42wKWEZEoIBnIa5Xo\n3DWjcYniRVV9rf5+VfzNTUAAAAO7SURBVC1W1f3e87lAtIiktVZ83nWzvJ85wOu46r6/YN7nUJsK\nLFPVvfV3tIX3ENhb1zTn/cwJUCas76OI3AhcBHzdS2iHCeJvISRUda+q1qqqD3i6geuG+/2LAq4A\nXm6oTLjev6N1LCWLxcAQERngffO8FphTr8wcoG7UyTTgw4b+o7Q0r33zH8A6VX24gTI96vpQRGQi\n7t+vNZNZgogk1j3HdYSurldsDvBNb1TUyUCRX5NLa2nwG12430OP/9/ZDcCbAcq8C5wvIqleM8v5\n3raQE5EpwI+BS1S1rIEywfwthCo+/z6wyxu4bjD/30PpXGC9qmb+//bu58WmMI7j+PuD8rP8KAoL\nGjYoKbLwY2VnZTFSmAU2ioWdhNT8A1ZTpiwMZkVsZDWzmJrFNCRGLJislLKRokh8LZ7v5fp5rsuc\nSz6vOjXz3Oeeec7Tc+Z773PO+T7fe7GT/de2Tl9hr3Oj3KnziHKXxMks66WcFACzKFMXk8A40FVj\n27ZRpiMmgLu57QQOA4ezzlHgAeXOjjFgS83915V/+162o9GHzW0U0Jd9fB/YVHMb51L++c9vKutY\nH1KC1jPgHWXe/BDlOtgw8BgYAhZl3U3A+ab3HsyxOAkcqLF9k5T5/sY4bNwhuAy4+bOxUFP7LuXY\nmqAEgKVfty9//+Z8r6N9WX6hMeaa6tbef39yc7oPMzOr9D9NQ5mZWZscLMzMrJKDhZmZVXKwMDOz\nSg4WZmZWycHC7C+Q2XBvdLodZj/iYGFmZpUcLMx+gaT9ksZzDYJ+SdMlvZJ0VmUdkmFJi7PuBklj\nTetCLMzy1ZKGMpnhHUmrcvfzJF3NtSQG68p4bNYKBwuzFklaA+wBtkbEBuA9sI/y1PjtiFgHjABn\n8i0XgeMRsZ7yxHGjfBDoi5LMcAvlCWAomYaPAWspT/hunfKDMmvRjE43wOwfsgPYCNzKD/2zKUkA\nP/A5Ydxl4Jqk+cCCiBjJ8gHgSuYDWh4R1wEi4g1A7m88MpdQrq62Ehid+sMyq+ZgYdY6AQMRceKL\nQun0V/XazaHztunn9/j8tL+Ip6HMWjcMdEtaAp/W0l5BOY+6s85eYDQiXgIvJG3P8h5gJMoqiE8l\n7cp9zJQ0p9ajMGuDP7mYtSgiHko6RVndbBol0+gR4DWwOV97TrmuASX9+LkMBk+AA1neA/RL6s19\n7K7xMMza4qyzZr9J0quImNfpdphNJU9DmZlZJX+zMDOzSv5mYWZmlRwszMyskoOFmZlVcrAwM7NK\nDhZmZlbpI3akh/A1KmftAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize History for Loss.\n",
    "plt.title('Keras model loss')\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()\n",
    "\n",
    "# Visualize History for Accuracy.\n",
    "plt.title('Keras model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D2L4TkncQb0F"
   },
   "source": [
    "Over time, loss decreases and accuracy increases. But do they converge to a\n",
    "stable level? Are there big differences between the training and validation\n",
    "metrics (a sign of overfitting)?\n",
    "\n",
    "Learn about [how to improve your machine learning\n",
    "model](https://developers.google.com/machine-learning/crash-course/). Then, feel\n",
    "free to adjust hyperparameters or the model architecture and train again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gmTiOQObmTqd"
   },
   "source": [
    "#### Export the model for serving\n",
    "\n",
    "AI Platform requires when you [create a model version\n",
    "resource](https://cloud.google.com/ml-engine/docs/tensorflow/deploying-models#create_a_model_version).\n",
    "\n",
    "Since not all optimizers can be exported to the SavedModel format, you may see\n",
    "warnings during the export process. As long you successfully export a serving\n",
    "graph, AI Platform can used the SavedModel to serve predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 479
    },
    "colab_type": "code",
    "id": "d4BaDUJzmTW_",
    "outputId": "7934b523-2892-4c0e-a3ab-b305ae247e4f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING:tensorflow:This model was compiled with a Keras optimizer (<tensorflow.python.keras.optimizers.RMSprop object at 0x7fc198c4e400>) but is being saved in TensorFlow format with `save_weights`. The model's weights will be saved, but unlike with TensorFlow optimizers in the TensorFlow format the optimizer's state will not be saved.\n",
      "\n",
      "Consider using a TensorFlow optimizer from `tf.train`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/network.py:1436: update_checkpoint_state (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.train.CheckpointManager to manage checkpoints rather than manually editing the Checkpoint proto.\n",
      "WARNING:tensorflow:Model was compiled with an optimizer, but the optimizer is not from `tf.train` (e.g. `tf.train.AdagradOptimizer`). Only the serving graph was exported. The train and evaluate graphs were not added to the SavedModel.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "INFO:tensorflow:No assets to save.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: keras_export/1553710367/saved_model.pb\n",
      "Model exported to:  b'keras_export/1553710367'\n"
     ]
    }
   ],
   "source": [
    "# Export the model to a local SavedModel directory \n",
    "export_path = tf.keras.experimental.export_saved_model(keras_model, 'keras_export')\n",
    "print(\"Model exported to: \", export_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dzGLPJbikj1t"
   },
   "source": [
    "You may export a SavedModel directory to your local filesystem or to Cloud\n",
    "Storage, as long as you have the necessary permissions. In your current\n",
    "environment, you granted access to Cloud Storage by authenticating your GCP account and setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.\n",
    "AI Platform training jobs can also export directly to Cloud Storage, because\n",
    "AI Platform service accounts [have access to Cloud Storage buckets in their own\n",
    "project](https://cloud.google.com/ml-engine/docs/tensorflow/working-with-cloud-storage).\n",
    "\n",
    "Try exporting directly to Cloud Storage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 258
    },
    "colab_type": "code",
    "id": "_eIF2qfykiFJ",
    "outputId": "d7ad6247-bd8b-4c67-b76d-71f5c3e91a0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:This model was compiled with a Keras optimizer (<tensorflow.python.keras.optimizers.RMSprop object at 0x7fc198c4e400>) but is being saved in TensorFlow format with `save_weights`. The model's weights will be saved, but unlike with TensorFlow optimizers in the TensorFlow format the optimizer's state will not be saved.\n",
      "\n",
      "Consider using a TensorFlow optimizer from `tf.train`.\n",
      "WARNING:tensorflow:Model was compiled with an optimizer, but the optimizer is not from `tf.train` (e.g. `tf.train.AdagradOptimizer`). Only the serving graph was exported. The train and evaluate graphs were not added to the SavedModel.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "INFO:tensorflow:No assets to save.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/saved_model.pb\n",
      "Model exported to:  b'gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379'\n"
     ]
    }
   ],
   "source": [
    "# Export the model to a SavedModel directory in Cloud Storage\n",
    "export_path = tf.keras.experimental.export_saved_model(keras_model, JOB_DIR + '/keras_export')\n",
    "print(\"Model exported to: \", export_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i805caqculq4"
   },
   "source": [
    "You can now deploy this model to AI Platform and serve predictions by\n",
    "following the steps from Part 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x27DXeUGzb-M"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all GCP resources used in this project, you can [delete the GCP\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Alternatively, you can clean up individual resources by running the following\n",
    "commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "colab_type": "code",
    "id": "no210oWF68Uk",
    "outputId": "b03da82b-41ac-4243-e2b9-b9b4be12b0e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/#1553709423098988...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/#1553709422852130...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/variables/checkpoint#1553709429999272...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/variables/#1553709428368512...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/assets/#1553709430948151...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/variables/variables.data-00000-of-00001#1553709428975750...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/assets/saved_model.json#1553709431121952...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/variables/variables.index#1553709429461522...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553709421/saved_model.pb#1553709430502605...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/#1553710381998179...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/assets/#1553710395035632...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/assets/saved_model.json#1553710395421499...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/saved_model.pb#1553710394249816...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/variables/#1553710390778836...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/variables/checkpoint#1553710393369087...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/variables/variables.data-00000-of-00001#1553710391609457...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/keras_export/1553710379/variables/variables.index#1553710392464814...\n",
      "Removing gs://<your-bucket-name>/keras-job-dir/packages/dcc159f40836cff74a27866227b327b0a8ccb5266194e76cff5368266b6d1cdd/trainer-0.0.0.tar.gz#1553709266664674...\n",
      "/ [18/18 objects] 100% Done                                                     \n",
      "Operation completed over 18 objects.                                             \n"
     ]
    }
   ],
   "source": [
    "# Delete model version resource\n",
    "! gcloud ai-platform versions delete $MODEL_VERSION --region $REGION --quiet --model $MODEL_NAME \n",
    "\n",
    "# Delete model resource\n",
    "! gcloud ai-platform models delete $MODEL_NAME --region $REGION --quiet\n",
    "\n",
    "# Delete Cloud Storage objects that were created\n",
    "! gsutil -m rm -r $JOB_DIR\n",
    "\n",
    "# If the training job is still running, cancel it\n",
    "! gcloud ai-platform jobs cancel $JOB_NAME --quiet --verbosity critical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3F2g4OjbJ3gZ"
   },
   "source": [
    "If your Cloud Storage bucket doesn't contain any other objects and you would like to delete it, run `gsutil rm -r gs://$BUCKET_NAME`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K0UXLWaBJnrY"
   },
   "source": [
    "## What's next?\n",
    "\n",
    "* View the [complete training\n",
    "code](https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census/tf-keras) used in this guide, which structures the code to accept custom\n",
    "hyperparameters as command-line flags.\n",
    "* Read about [packaging\n",
    "code](https://cloud.google.com/ml-engine/docs/tensorflow/packaging-trainer) for an AI Platform training job.\n",
    "* Read about [deploying a\n",
    "model](https://cloud.google.com/ml-engine/docs/tensorflow/deploying-models) to serve predictions."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "getting-started-keras.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}