notebooks/official/prediction/get_started_with_vertex_private_endpoints.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# Get started with Vertex AI Private Endpoints\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_vertex_private_endpoints.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_vertex_private_endpoints.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/prediction/get_started_with_vertex_private_endpoints.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>          \n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI SDK to create and use `Vertex AI Endpoint` resources for serving models. `Vertex AI Private Endpoints`. A `Private Endpoint` provides peer-to-peer network gRPC communication (i.e., intranet) between client and server, within the same network. This eliminates the overhead of networking switching and routing of a public endpoint which uses the HTTP protocol (i.e., internet).\n",
        "\n",
        "Learn more about [Private Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/using-private-endpoints).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9402cfbdc2d"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use `Vertex AI Private Endpoint` resources.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- `Vertex AI Endpoints`\n",
        "- `Vertex AI Models`\n",
        "- `Vertex AI Prediction`\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Creating a `Private Endpoint` resource.\n",
        "- Configure a VPC peering connection.\n",
        "- Configuring the serving binary of a `Model` resource for deployment to a `Private Endpoint` resource.\n",
        "- Deploying a `Model` resource to a `Private Endpoint` resource.\n",
        "- Send a prediction request to a `Private Endpoint`\n",
        "\n",
        "#### How does `Private Endpoint` setup differ from `Public Endpoint`\n",
        "\n",
        "- Enable two additional APIs: Service Networking and Cloud DNS.\n",
        "- Add Compute Admin Network role to your (default) service account.\n",
        "- Issue two gcloud commands to setup the VPC peering for your service account.\n",
        "- There is *currently* no SDK support yet, so private endpoint is created with GAPIC client and has an extra argument for the peering network.\n",
        "- To send a request, you can't use SDK/GAPIC since they do a HTTP internet request. Instead, you use curl to send a peer-to-peer request."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:iris,lcn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.\n",
        "\n",
        "Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform\n",
        "! pip3 install --upgrade --quiet google-cloud-pipeline-components\n",
        "! pip3 install tensorflow-hub\n",
        "! pip3 install --upgrade --quiet tensorflow "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### GPU runtime\n",
        "\n",
        "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage-component.googleapis.com)\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
        "\n",
        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "project_id"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
        "\n",
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated.\n",
        "\n",
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "254614fa0c46"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a336a05c6149"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6b2ccc891ed"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {REGION} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "import google.cloud.aiplatform as aip\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for prediction.\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2.5\".replace(\".\", \"-\")\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    REGION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for prediction.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you will use for for prediction.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from TensorFlow Hub\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a `Vertex AI Model` resource. Once you have a `Vertex AI Model` resource, the model can be deployed to a `Vertex AI Private Endpoint` resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "tfhub_model = tf.keras.Sequential(\n",
        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
        ")\n",
        "\n",
        "tfhub_model.build([None, 224, 224, 3])\n",
        "\n",
        "tfhub_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63de49055083"
      },
      "source": [
        "### Save the model artifacts\n",
        "\n",
        "At this point, the model is in memory. Next, you save the model artifacts to a Cloud Storage location."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64618c713db9"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
        "tfhub_model.save(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "how_serving_function_works"
      },
      "source": [
        "## Upload the model for serving\n",
        "\n",
        "Next, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex AI, your serving function ensures that the data is decoded on the model server before it is passed as input to your model.\n",
        "\n",
        "### How does the serving function work\n",
        "\n",
        "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n",
        "\n",
        "The serving function consists of two parts:\n",
        "\n",
        "- `preprocessing function`:\n",
        "  - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n",
        "  - Performs the same preprocessing of the data that was done during training the underlying model -- e.g., normalizing, scaling, etc.\n",
        "- `post-processing function`:\n",
        "  - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n",
        "  - Packages the output for the the receiving application -- e.g., add headings, make JSON object, etc.\n",
        "\n",
        "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n",
        "\n",
        "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_image:post"
      },
      "source": [
        "### Serving function for image data\n",
        "\n",
        "#### Preprocessing\n",
        "\n",
        "To pass images to the prediction service, you encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes, and then preprocessed to match the model input requirements, before it is passed as input to the deployed model.\n",
        "\n",
        "To resolve this, you define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is fused to the underlying model (instead of upstream on a CPU).\n",
        "\n",
        "When you send a prediction or explanation request, the content of the request is base 64 decoded into a Tensorflow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:\n",
        "\n",
        "- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).\n",
        "- `image.convert_image_dtype` - Changes integer pixel values to float 32, and rescales pixel data between 0 and 1.\n",
        "- `image.resize` - Resizes the image to match the input shape for the model.\n",
        "\n",
        "At this point, the data can be passed to the model (`m_call`), via a concrete function. The serving function is a static graph, while the model is a dynamic graph. The concrete function performs the tasks of marshalling the input data from the serving function to the model, and marshalling the prediction result from the model back to the serving function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_image"
      },
      "outputs": [],
      "source": [
        "CONCRETE_INPUT = \"numpy_inputs\"\n",
        "\n",
        "\n",
        "def _preprocess(bytes_input):\n",
        "    decoded = tf.io.decode_jpeg(bytes_input, channels=3)\n",
        "    decoded = tf.image.convert_image_dtype(decoded, tf.float32)\n",
        "    resized = tf.image.resize(decoded, size=(224, 224))\n",
        "    return resized\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def preprocess_fn(bytes_inputs):\n",
        "    decoded_images = tf.map_fn(\n",
        "        _preprocess, bytes_inputs, dtype=tf.float32, back_prop=False\n",
        "    )\n",
        "    return {\n",
        "        CONCRETE_INPUT: decoded_images\n",
        "    }  # User needs to make sure the key matches model's input\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def serving_fn(bytes_inputs):\n",
        "    images = preprocess_fn(bytes_inputs)\n",
        "    prob = m_call(**images)\n",
        "    return prob\n",
        "\n",
        "\n",
        "m_call = tf.function(tfhub_model.call).get_concrete_function(\n",
        "    [tf.TensorSpec(shape=[None, 224, 224, 3], dtype=tf.float32, name=CONCRETE_INPUT)]\n",
        ")\n",
        "\n",
        "tf.saved_model.save(tfhub_model, MODEL_DIR, signatures={\"serving_default\": serving_fn})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "source": [
        "## Get the serving function signature\n",
        "\n",
        "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n",
        "\n",
        "For your purpose, you need the signature of the serving function. Why? Well, when we send our data for prediction as a HTTP request packet, the image data is base64 encoded, and our TF.Keras model takes numpy input. Your serving function will do the conversion from base64 to a numpy array.\n",
        "\n",
        "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "outputs": [],
      "source": [
        "loaded = tf.saved_model.load(MODEL_DIR)\n",
        "\n",
        "serving_input = list(\n",
        "    loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n",
        ")[0]\n",
        "print(\"Serving function input:\", serving_input)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "### Upload the TensorFlow Hub model to a `Vertex AI Model` resource\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model into a `Vertex AI Model` resource."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "model = aip.Model.upload(\n",
        "    display_name=\"example\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        ")\n",
        "\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4f4f0bc64ddb"
      },
      "source": [
        "## Setup VPC peering network\n",
        "\n",
        "To use a `Private Endpoint`, you setup a VPC peering network between your project and the `Vertex AI Prediction` service project that is hosting VMs running your model. This eliminates additional hops in network traffic and allows using efficient gRPC protocol.\n",
        "\n",
        "Learn more about [VPC peering](https://cloud.google.com/vertex-ai/docs/general/vpc-peering).\n",
        "\n",
        "**IMPORTANT: you can only setup one VPC peering to servicenetworking.googleapis.com per project.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d85e8f48291a"
      },
      "source": [
        "### Create VPC peering for default network\n",
        "\n",
        "For simplicity, we setup VPC peering to the default network. You can create a different network for your project.\n",
        "\n",
        "If you setup VPC peering with any other network, make sure that the network already exists and that your VM is running on that network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a107544fbabf"
      },
      "outputs": [],
      "source": [
        "# This is for display only; you can name the range anything.\n",
        "PEERING_RANGE_NAME = \"vertex-ai-prediction-peering-range\"\n",
        "NETWORK = \"default\"\n",
        "\n",
        "# NOTE: `prefix-length=16` means a CIDR block with mask /16 will be\n",
        "# reserved for use by Google services, such as Vertex AI.\n",
        "! gcloud compute addresses create $PEERING_RANGE_NAME \\\n",
        "  --global \\\n",
        "  --prefix-length=16 \\\n",
        "  --description=\"peering range for Google service\" \\\n",
        "  --network=$NETWORK \\\n",
        "  --purpose=VPC_PEERING"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8e29cad1a0be"
      },
      "source": [
        "### Create the VPC connection\n",
        "\n",
        "Next, create the connection for VPC peering.\n",
        "\n",
        "*Note:* If you get a PERMISSION DENIED, you may not have the neccessary role 'Compute Network Admin' set for your default service account. In the Cloud Console, do the following steps.\n",
        "\n",
        "1. Goto `IAM & Admin`\n",
        "2. Find your service account.\n",
        "3. Click edit icon.\n",
        "4. Select `Add Another Role`.\n",
        "5. Enter 'Compute Network Admin'.\n",
        "6. Select `Save`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f3f6c85ffc63"
      },
      "outputs": [],
      "source": [
        "! gcloud services vpc-peerings connect \\\n",
        "  --service=servicenetworking.googleapis.com \\\n",
        "  --network=$NETWORK \\\n",
        "  --ranges=$PEERING_RANGE_NAME \\\n",
        "  --project=$PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "944d772b1397"
      },
      "source": [
        "Check the status of your peering connections."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b946ce37cc16"
      },
      "outputs": [],
      "source": [
        "! gcloud compute networks peerings list --network $NETWORK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a5e1b83ae61"
      },
      "source": [
        "#### Construct the full network name\n",
        "\n",
        "You need to have the full network resource name when you subsequently create an `Private Endpoint` resource for VPC peering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cd58eb809f71"
      },
      "outputs": [],
      "source": [
        "project_number = model.resource_name.split(\"/\")[1]\n",
        "print(project_number)\n",
        "\n",
        "full_network_name = f\"projects/{project_number}/global/networks/{NETWORK}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "628de0914ba1"
      },
      "source": [
        "## Creating an `Endpoint` resource\n",
        "\n",
        "You create an `Endpoint` resource using the `PrivateEndpoint.create()` method. \n",
        "\n",
        "In this example, the following parameters are specified:\n",
        "\n",
        "- `display_name`: A human readable name for the `Private Endpoint` resource.\n",
        "- `network`: The full network resource name for the VPC peering.\n",
        "\n",
        "Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bb1fe4502e92"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    endpoint = aip.PrivateEndpoint.create(\n",
        "        display_name=\"private\", network=full_network_name\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "15d2ebdb1a03"
      },
      "source": [
        "### Get details on an `Endpoint` resource\n",
        "\n",
        "You can get the underlying details of an `Endpoint` object with the property `gca_resource`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ddb9410f2570"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    endpoint.gca_resource"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca3fa3f6a894"
      },
      "source": [
        "## Deploying `Model` resources to an `Endpoint` resource.\n",
        "\n",
        "You can deploy one of more `Vertex AI Model` resource instances to the same endpoint. Each `Vertex AI Model` resource that is deployed will have its own deployment container for the serving binary. \n",
        "\n",
        "*Note:* For this example, you specified the deployment container for the TFHub model in the previous step of uploading the model artifacts to a `Vertex AI Model` resource.\n",
        "\n",
        "### Deploying a single `Endpoint` resource\n",
        "\n",
        "In the next example, you deploy a single `Vertex AI Model` resource to a `Vertex AI Endpoint` resource. To deploy, you specify the following additional configuration settings:\n",
        "\n",
        "- The machine type.\n",
        "- The (if any) type and number of GPUs.\n",
        "- Static, manual or auto-scaling of VM instances.\n",
        "\n",
        "For a `Private Endpoint`, you can only deploy a single model. As such, there is no traffic split. \n",
        "\n",
        "In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
        "\n",
        "- `model`: The `Model` resource.\n",
        "- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
        "- `machine_type`: The machine type for each VM instance.\n",
        "\n",
        "Do to the requirements to provision the resource, this may take upto a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4e93b034a72f"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    response = endpoint.deploy(\n",
        "        model=model,\n",
        "        deployed_model_display_name=\"example_\" + TIMESTAMP,\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "    )\n",
        "\n",
        "    print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1ae5a228adb"
      },
      "source": [
        "#### Get information on the deployed model\n",
        "\n",
        "You can get the deployment settings of the deployed model from the `Endpoint` resource configuration data `gca_resource.deployed_models`. In this example, only one model is deployed -- hence the reference to the subscript `[0]`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5864deb1fd90"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    print(endpoint.gca_resource.deployed_models[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "daf4494bd342"
      },
      "source": [
        "### Prepare test data for prediction\n",
        "\n",
        "Next, you will load a compressed JPEG image into memory and then base64 encode it. For demonstration purposes, you use an image from the Flowers dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "486dc48ce249"
      },
      "outputs": [],
      "source": [
        "! gsutil cp gs://cloud-ml-data/img/flower_photos/daisy/100080576_f52e8ee070_n.jpg test.jpg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9cde1862983"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "\n",
        "with open(\"test.jpg\", \"rb\") as f:\n",
        "    data = f.read()\n",
        "b64str = base64.b64encode(data).decode(\"utf-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "source": [
        "### Make the prediction\n",
        "\n",
        "Now that your `Model` resource is deployed to an `Endpoint` resource, you can do online predictions by sending prediction requests to the Endpoint resource.\n",
        "\n",
        "#### Request\n",
        "\n",
        "Since a `Private Endpoint` would block a request from a public HTTP request, you will send the request using `curl` to a private URI.\n",
        "\n",
        "First, you will construct the request as a JSON file. To pass the test data to the prediction service, you encode the bytes into base64 -- which makes the content safe from modification while transmitting binary data over the network.\n",
        "\n",
        "The format of each instance is:\n",
        "\n",
        "    { serving_input: { 'b64': base64_encoded_bytes } }\n",
        "\n",
        "Since the serving binary can take multiple items (instances), send your single test item as a list of one test item."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9e660cfcc51d"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "with open(\"instances.json\", \"w\") as f:\n",
        "    f.write(json.dumps({\"instances\": [{serving_input: {\"b64\": b64str}}]}))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef4b9c97ab71"
      },
      "source": [
        "### Make the prediction request using SDK\n",
        "\n",
        "Next, use the `Vertex AI SDK` to make a prediction request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e75c3ddd44e6"
      },
      "outputs": [],
      "source": [
        "instances = [{serving_input: {\"b64\": b64str}}]\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    prediction = endpoint.predict(instances=instances)\n",
        "    print(prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cf4e3ce528b"
      },
      "source": [
        "### Undeploy `Model` resource from `Endpoint` resource\n",
        "\n",
        "When a `Model` resource is deployed to an `Endpoint` resource, the deployed `Model` resource instance is assigned an ID -- commonly referred to as the deployed model ID.\n",
        "\n",
        "You can undeploy a specific `Model` resource instance with the `undeploy()` method, with the following parameters:\n",
        "\n",
        "- `deployed_model_id`: The ID assigned to the deployed model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "69557a907de7"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    deployed_model_id = endpoint.gca_resource.deployed_models[0].id\n",
        "    print(deployed_model_id)\n",
        "\n",
        "    endpoint.undeploy(deployed_model_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d89e913932a"
      },
      "source": [
        "### Undeploy all `Model` resources from an `Endpoint` resource.\n",
        "\n",
        "Finally, you can undeploy all `Model` instances from an `Endpoint` resource using the `undeploy_all()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "09ff279b71ec"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    endpoint.undeploy_all()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "64b75c0a6f8e"
      },
      "source": [
        "### Delete an `Endpoint` resource\n",
        "\n",
        "You can delete an `Endpoint` resource with the `delete()` method if the `Endpoint` resource has no deployed models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d54790378636"
      },
      "outputs": [],
      "source": [
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    endpoint.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "delete_bucket = False\n",
        "delete_model = True\n",
        "\n",
        "if delete_model:\n",
        "    try:\n",
        "        model.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil rm -rf {BUCKET_URI}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "get_started_with_vertex_private_endpoints.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}