notebooks/tensorflow/custom-prediction-routine-keras.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KXNKiFZTUySu"
   },
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "137G11XIWgCy"
   },
   "source": [
    "# Creating a custom prediction routine with Keras\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routine-keras\">\n",
    "      <img src=\"https://cloud.google.com/_static/images/cloud/icons/favicons/onecloud/super_cloud.png\"\n",
    "           alt=\"Google Cloud logo\" width=\"32px\"> Read on cloud.google.com\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/tensorflow/custom-prediction-routine-keras.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/cloudml-samples/blob/main/notebooks/tensorflow/custom-prediction-routine-keras.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KXEex-vKeSkC"
   },
   "source": [
    "**Beta**\n",
    "\n",
    "This is a beta release of custom prediction routines. This feature might be changed in backward-incompatible ways and is not subject to any SLA or deprecation policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tW9YCwlGY63D"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This tutorial shows how to deploy a trained Keras model to AI Platform and serve predictions using a [custom prediction routine](https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routines). This lets you customize how AI Platform responds to each prediction request.\n",
    "\n",
    "In this example, you will use a custom prediction routine to preprocess\n",
    "prediction input by scaling it, and to postprocess prediction output by converting [softmax](https://developers.google.com/machine-learning/glossary/#s) probability outputs to label strings.\n",
    "\n",
    "The tutorial walks through several steps:\n",
    "\n",
    "* Training a simple Keras model locally (in this notebook)\n",
    "* Creating and deploy a custom prediction routine to AI Platform\n",
    "* Serving prediction requests from that deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pd0wS_kKY_Du"
   },
   "source": [
    "### Dataset\n",
    "\n",
    "This tutorial uses R.A. Fisher's Iris dataset, a small dataset that is popular for trying out machine learning techniques. Each instance has four numerical features, which are different measurements of a flower, and a target label that\n",
    "marks it as one of three types of iris: Iris setosa, Iris versicolour, or Iris virginica.\n",
    "\n",
    "This tutorial uses [the copy of the Iris dataset included in the\n",
    "scikit-learn library](https://scikit-learn.org/stable/datasets/index.html#iris-dataset)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2OFY1EMTZAfI"
   },
   "source": [
    "### Objective\n",
    "\n",
    "The goal is to train a model that uses a flower's measurements as input to predict what type of iris it is.\n",
    "\n",
    "This tutorial focuses more on using this model with AI Platform than on\n",
    "the design of the model itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GIU-iw6WZRJO"
   },
   "source": [
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
    "\n",
    "* AI Platform\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [AI Platform\n",
    "pricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "91gDvdNHZX5S"
   },
   "source": [
    "## Before you begin\n",
    "\n",
    "You must do several things before you can train and deploy a model in AI Platform:\n",
    "\n",
    "* Set up your local development environment.\n",
    "* Set up a GCP project with billing and the necessary\n",
    "  APIs enabled.\n",
    "* Authenticate your GCP account in this notebook.\n",
    "* Create a Cloud Storage bucket to store your training package and your\n",
    "  trained model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VC2c7_HJZZT_"
   },
   "source": [
    "### Set up your local development environment\n",
    "\n",
    "**If you are using Colab or AI Platform Notebooks**, your environment already\n",
    "meets all the requirements to run this notebook. You can skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VwDsbQE0a0Mp"
   },
   "source": [
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "3. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3.\n",
    "\n",
    "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
    "   Jupyter.\n",
    "\n",
    "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
    "\n",
    "6. Open this notebook in the Jupyter Notebook Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ldy2Z_z-a-0z"
   },
   "source": [
    "### Set up your GCP project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. [Enable the AI Platform (\"Cloud Machine Learning Engine\") and Compute Engine\n",
    "APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
    "\n",
    "4. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wsuRZACjbiES"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"<your-project-id>\" #@param {type:\"string\"}\n",
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x9hqlEa-bpIX"
   },
   "source": [
    "### Authenticate your GCP account\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QvekLkQibra5"
   },
   "source": [
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **Machine Learning Engine > AI Platform Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment.\n",
    "\n",
    "6. Enter the path to your service account key as the\n",
    "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qkL641qHb1SZ"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your GCP account. This provides access to your\n",
    "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "if 'google.colab' in sys.modules:\n",
    "  from google.colab import auth as google_auth\n",
    "  google_auth.authenticate_user()\n",
    "\n",
    "# If you are running this notebook locally, replace the string below with the\n",
    "# path to your service account key and run this cell to authenticate your GCP\n",
    "# account.\n",
    "else:\n",
    "  %env GOOGLE_APPLICATION_CREDENTIALS '<path-to-your-service-account-key.json>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O4E2pjYacgtz"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "To deploy a custom prediction routine, you must upload your trained model\n",
    "artifacts and your custom code to Cloud Storage.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets. \n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
    "AI Platform services are\n",
    "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions). You may\n",
    "not use a Multi-Regional Storage bucket for training with AI Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "31wtdj9od-3R"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"<your-bucket-name>\" #@param {type:\"string\"}\n",
    "REGION = \"us-central1\" #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3EG9EzUOeBPI"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HWLSNArseDWH"
   },
   "outputs": [],
   "source": [
    "! gsutil mb -l $REGION gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BjW8HyEDeGlM"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XpORnbjPeMpK"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NKQvwF34NaVO"
   },
   "source": [
    "## Building and training a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i6p0xz9f6PN2"
   },
   "source": [
    "Often, you can't use your data in its raw form to train a machine learning model. Even when you can, preprocessing the data before using it for training can sometimes improve your model.\n",
    "\n",
    "Assuming that you expect the input for prediction to have the same format as your training data, you must apply identical preprocessing during training and prediction to ensure that your model makes consistent predictions.\n",
    "\n",
    "In this section, create a preprocessing module and use it as part of training. Then export a preprocessor with characteristics learned during training to use later in your custom prediction routine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D2yQRtJ-ZUi_"
   },
   "source": [
    "### Install dependencies for local training\n",
    "\n",
    "Training locally (in the notebook) requires several dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-PZYdYeUZT2-"
   },
   "outputs": [],
   "source": [
    "! pip3 install numpy scikit-learn 'tensorflow>=1.13,<2'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K2tWqy3i-dtA"
   },
   "source": [
    "### Write your preprocessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RFf2nh4x-iRL"
   },
   "source": [
    "Scaling training data so each numerical feature column has a mean of 0 and a standard deviation of 1 [can improve your model](https://developers.google.com/machine-learning/crash-course/representation/cleaning-data).\n",
    "\n",
    "Create `preprocess.py`, which contains a class to do this scaling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NLTUnqaG_Elo"
   },
   "outputs": [],
   "source": [
    "%%writefile preprocess.py\n",
    "import numpy as np\n",
    "\n",
    "class MySimpleScaler(object):\n",
    "  def __init__(self):\n",
    "    self._means = None\n",
    "    self._stds = None\n",
    "\n",
    "  def preprocess(self, data):\n",
    "    if self._means is None: # during training only\n",
    "      self._means = np.mean(data, axis=0)\n",
    "\n",
    "    if self._stds is None: # during training only\n",
    "      self._stds = np.std(data, axis=0)\n",
    "      if not self._stds.all():\n",
    "        raise ValueError('At least one column has standard deviation of 0.')\n",
    "\n",
    "    return (data - self._means) / self._stds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sF5n_B68_1PO"
   },
   "source": [
    "Notice that an instance of `MySimpleScaler` saves the means and standard deviations of each feature column on first use. Then it uses these summary statistics to scale data it encounters afterward.\n",
    "\n",
    "This lets you store characteristics of the training distribution and use them for identical preprocessing at prediction time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i-gV-88fAmXN"
   },
   "source": [
    "### Train your model\n",
    "\n",
    "Next, use `preprocess.MySimpleScaler` to preprocess the iris data, then train a simple neural network using Keras.\n",
    "\n",
    "At the end, export your trained Keras model as an HDF5 (`.h5`) file and export\n",
    "your `MySimpleScaler` instance as a pickle (`.pkl`) file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OAARevWVCBRw"
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "import tensorflow as tf\n",
    "\n",
    "from preprocess import MySimpleScaler\n",
    "\n",
    "iris = load_iris()\n",
    "scaler = MySimpleScaler()\n",
    "num_classes = len(iris.target_names)\n",
    "X = scaler.preprocess(iris.data)\n",
    "y = tf.keras.utils.to_categorical(iris.target, num_classes=num_classes)\n",
    "\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Dense(25, activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.Dense(25, activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.Dense(num_classes, activation=tf.nn.softmax))\n",
    "model.compile(\n",
    "  optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(X, y, epochs=10, batch_size=1)\n",
    "\n",
    "model.save('model.h5')\n",
    "with open ('preprocessor.pkl', 'wb') as f:\n",
    "  pickle.dump(scaler, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GSuZ1M2_LuVm"
   },
   "source": [
    "Note: When deploying a TensorFlow model to AI Platform without a custom prediction routine, you must export the trained model in the `SavedModel` format. When you deploy a custom prediction routine, you are able to export to the HDF5 format instead—or any other format that suits your needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KsQ7Vnm7PEw3"
   },
   "source": [
    "## Deploying a custom prediction routine\n",
    "\n",
    "To deploy a custom prediction routine to serve predictions from your trained model, do the following:\n",
    "\n",
    "* Create a custom predictor to handle requests\n",
    "* Package your predictor and your preprocessing module\n",
    "* Upload your model artifacts and your custom code to Cloud Storage\n",
    "* Deploy your custom prediction routine to AI Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OxPi36JRiUPy"
   },
   "source": [
    "### Create a custom predictor\n",
    "\n",
    "To deploy a custom prediction routine, you must create a class that implements\n",
    "the [Predictor interface](https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routines#predictor-class). This tells AI Platform how to load your model and how to handle prediction requests.\n",
    "\n",
    "Write the following code to `predictor.py`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0DtxCmp1uz7C"
   },
   "outputs": [],
   "source": [
    "%%writefile predictor.py\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_iris\n",
    "import tensorflow as tf\n",
    "\n",
    "class MyPredictor(object):\n",
    "  def __init__(self, model, preprocessor):\n",
    "    self._model = model\n",
    "    self._preprocessor = preprocessor\n",
    "    self._class_names = load_iris().target_names\n",
    "\n",
    "  def predict(self, instances, **kwargs):\n",
    "    inputs = np.asarray(instances)\n",
    "    preprocessed_inputs = self._preprocessor.preprocess(inputs)\n",
    "    outputs = self._model.predict(preprocessed_inputs)\n",
    "    if kwargs.get('probabilities'):\n",
    "      return outputs.tolist()\n",
    "    else:\n",
    "      return [self._class_names[index] for index in np.argmax(outputs, axis=1)]\n",
    "\n",
    "  @classmethod\n",
    "  def from_path(cls, model_dir):\n",
    "    model_path = os.path.join(model_dir, 'model.h5')\n",
    "    model = tf.keras.models.load_model(model_path)\n",
    "\n",
    "    preprocessor_path = os.path.join(model_dir, 'preprocessor.pkl')\n",
    "    with open(preprocessor_path, 'rb') as f:\n",
    "      preprocessor = pickle.load(f)\n",
    "\n",
    "    return cls(model, preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nmxYGrTXWV8k"
   },
   "source": [
    "Notice that, in addition to using the preprocessor that you defined during training, this predictor performs a postprocessing step that converts the neural network's softmax output (an array denoting the probability of each label being the correct one) into the label with the highest probability.\n",
    "\n",
    "However, if the predictor receives a `probabilities` keyword argument with the value `True`, it\n",
    "returns the probability array instead. The last part of this tutorial shows how to provide this keyword argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2lTNSrN_Pp-H"
   },
   "source": [
    "### Package your custom code\n",
    "\n",
    "You must package `predictor.py` and `preprocess.py` as a `.tar.gz` source distribution package and provide the package to AI Platform so it can use your custom code to serve predictions.\n",
    "\n",
    "Write the following `setup.py` to define your package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jc3N0McbQf6p"
   },
   "outputs": [],
   "source": [
    "%%writefile setup.py\n",
    "from setuptools import setup\n",
    "\n",
    "setup(\n",
    "    name='my_custom_code',\n",
    "    version='0.1',\n",
    "    scripts=['predictor.py', 'preprocess.py'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hUGh12uqQiTe"
   },
   "source": [
    "Then run the following command to create`dist/my_custom_code-0.1.tar.gz`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gHTU8TY5QvPe"
   },
   "outputs": [],
   "source": [
    "! python setup.py sdist --formats=gztar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kcoqfmh2QzSx"
   },
   "source": [
    "### Upload model artifacts and custom code to Cloud Storage\n",
    "\n",
    "Before you can deploy your model for serving, AI Platform needs access to the following files in Cloud Storage:\n",
    "\n",
    "* `model.h5` (model artifact)\n",
    "* `preprocessor.pkl` (model artifact)\n",
    "* `my_custom_code-0.1.tar.gz` (custom code)\n",
    "\n",
    "Model artifacts must be stored together in a model directory, which your\n",
    "Predictor can access as [the `model_dir` argument in its `from_path` class\n",
    "method](https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routines#predictor-class). The custom\n",
    "code does not need to be in the same directory. Run the following commands to\n",
    "upload your files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lYKvmNBcSD1k"
   },
   "outputs": [],
   "source": [
    "! gsutil cp ./dist/my_custom_code-0.1.tar.gz gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz\n",
    "! gsutil cp model.h5 preprocessor.pkl gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SLx50raLTIoc"
   },
   "source": [
    "### Deploy your custom prediction routine\n",
    "\n",
    "Create [a *model* resource and a *version* resource](https://cloud.google.com/ml-engine/docs/tensorflow/projects-models-versions-jobs) to deploy your custom prediction routine. First define variables with your resource names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hDvyNg5KTYoe"
   },
   "outputs": [],
   "source": [
    "MODEL_NAME = 'IrisPredictor'\n",
    "VERSION_NAME = 'v1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7mNzQf6YTj2g"
   },
   "source": [
    "Please note that currently custom prediction routine is only available on global endpoint. By specifying `--regions` flag in `gcloud ai-platform models create` command, you will use global endpoint.\n",
    "\n",
    "Then create your model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QsknxFOFT1PR"
   },
   "outputs": [],
   "source": [
    "! gcloud ai-platform models create $MODEL_NAME \\\n",
    "  --regions $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5Q0-ZM2ST8Z6"
   },
   "source": [
    "For all other gcloud command except for `gcloud ai-platform models create`, you need to specify `--region global` to use global endpoint. If you do not specify `--region` flag, there will be a prompt to let you choose an endpoint.\n",
    "\n",
    "Next, create a version. In this step, provide paths to the artifacts and custom code you uploaded to Cloud Storage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xK4sTvcuUVNg"
   },
   "outputs": [],
   "source": [
    "# --quiet automatically installs the beta component if it isn't already installed \n",
    "! gcloud --quiet beta ai-platform versions create $VERSION_NAME \\\n",
    "  --region global \\\n",
    "  --model $MODEL_NAME \\\n",
    "  --runtime-version 1.13 \\\n",
    "  --python-version 3.5 \\\n",
    "  --origin gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/ \\\n",
    "  --package-uris gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz \\\n",
    "  --prediction-class predictor.MyPredictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NthWBP3kUvX6"
   },
   "source": [
    "Learn more about [the options you must specify](https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routines#deploy_your_custom_prediction_routine) when you deploy a custom prediction routine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "soCVZR1H8mQK"
   },
   "source": [
    "## Serving online predictions\n",
    "\n",
    "Try out your deployment by sending an online prediction request. First, install the Google APIs Client Library for Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2SGwfntsrEYs"
   },
   "outputs": [],
   "source": [
    "! pip install --upgrade google-api-python-client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SJyorPp3rEAw"
   },
   "source": [
    "Then send two instances of iris data to your deployed version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cSt7ol37_FuB"
   },
   "outputs": [],
   "source": [
    "import googleapiclient.discovery\n",
    "\n",
    "instances = [\n",
    "  [6.7, 3.1, 4.7, 1.5],\n",
    "  [4.6, 3.1, 1.5, 0.2],\n",
    "]\n",
    "\n",
    "service = googleapiclient.discovery.build('ml', 'v1')\n",
    "name = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, VERSION_NAME)\n",
    "\n",
    "response = service.projects().predict(\n",
    "    name=name,\n",
    "    body={'instances': instances}\n",
    ").execute()\n",
    "\n",
    "if 'error' in response:\n",
    "    raise RuntimeError(response['error'])\n",
    "else:\n",
    "  print(response['predictions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PzwzYgoV-Bjk"
   },
   "source": [
    "Note: This code uses the credentials you set up during the authentication step to make the online prediction request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QYKzR1VkXTWy"
   },
   "source": [
    "### Sending keyword arguments\n",
    "\n",
    "When you send a prediction request to a custom prediction routine, you can provide [additional fields on your request body](https://cloud.google.com/ml-engine/docs/v1/predict-request). The Predictor's `predict` method receives these as fields of the `**kwargs` dictionary.\n",
    "\n",
    "The following code sends the same request as before, but this time it adds a `probabilities` field to the request body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9zYA_dZcX_KQ"
   },
   "outputs": [],
   "source": [
    "response = service.projects().predict(\n",
    "    name=name,\n",
    "    body={'instances': instances, 'probabilities': True}\n",
    ").execute()\n",
    "\n",
    "if 'error' in response:\n",
    "    raise RuntimeError(response['error'])\n",
    "else:\n",
    "  print(response['predictions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kLX2V3TyswzH"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all GCP resources used in this project, you can [delete the GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Alternatively, you can clean up individual resources by running the following commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ToGeA0Htssrl"
   },
   "outputs": [],
   "source": [
    "# Delete version resource\n",
    "! gcloud ai-platform versions delete $VERSION_NAME --region global --quiet --model $MODEL_NAME \n",
    "\n",
    "# Delete model resource\n",
    "! gcloud ai-platform models delete $MODEL_NAME --region global --quiet\n",
    "\n",
    "# Delete Cloud Storage objects that were created\n",
    "! gsutil -m rm -r gs://$BUCKET_NAME/custom_prediction_routine_tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i3inFBtMddy3"
   },
   "source": [
    "## What's next\n",
    "\n",
    "* Read more about [custom prediction routines](https://cloud.google.com/ml-engine/docs/tensorflow/custom-prediction-routines).\n",
    "* Learn about how to [export a model](https://cloud.google.com/ml-engine/docs/tensorflow/exporting-for-prediction) to serve predictions without a custom prediction routine."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Creating a custom prediction routine with Keras",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}