notebooks/official/training/hyperparameter_tuning_tensorflow.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Run hyperparameter tuning for a TensorFlow model\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/hyperparameter_tuning_tensorflow.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Ftraining%2Fhyperparameter_tuning_tensorflow.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>  \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/training/hyperparameter_tuning_tensorflow.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/hyperparameter_tuning_tensorflow.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>                                                                                        \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "The values you select for your model’s hyperparameters can make all the difference. If you’re only trying to tune a handful of hyperparameters, you might be able to run experiments manually. But when you start juggling hyperparameters for your model’s architecture, the optimizer, and finding the best batch size and learning rate, automating these experiments at scale quickly becomes a necessity. \n",
        "\n",
        "And it’s not just about tracking the results from all these trials. You also want a way to efficiently search the space of possible values so you don’t waste as much time trying out combinations that yield low accuracy scores.\n",
        "\n",
        "Vertex AI Training includes a hyperparameter tuning service. A Vertex AI hyperparameter tuning job runs multiple trials of your training code. On each trial, it uses different values for your chosen hyperparameters, set within the limits you specify. By default, the service uses Bayesian optimization to search the space of possible hyperparameter values. This means that information from prior experiments is used to select the next set of values, making the search more efficient. \n",
        "\n",
        "Learn more about [Vertex AI Hyperparameter Tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to run a Vertex AI hyperparameter tuning job for a TensorFlow model. While this example uses TensorFlow, you can also use this service for other ML frameworks.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "* Vertex AI Training\n",
        "* Cloud Storage\n",
        "* Artifact Registry\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "* Modify training application code for automated hyperparameter tuning.\n",
        "* Containerize training application code.\n",
        "* Configure and launch a hyperparameter tuning job with the Vertex AI Python SDK."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This sample uses the [Horses or Humans dataset](https://www.tensorflow.org/datasets/catalog/horses_or_humans) available through [TensorFlow datasets](https://www.tensorflow.org/datasets) to train a binary image classification model (horse or human)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "* Artifact Registry\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "[Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
        "and [Artifact Registry pricing](https://cloud.google.com/artifact-registry/pricing)\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7cc240e9c1a6"
      },
      "source": [
        "## Get Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "972161a99004"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff666ce4051c"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc7251520a07"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "496fcb0092a7"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}\n",
        "\n",
        "LOCATION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cc5ef354b46e"
      },
      "source": [
        "### UUID\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, create a uuid for each instance session, and append it onto the name of resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1105933b5528"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of a specifed length(default=8)\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2e29a9eb385"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "378ffacc0348"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aiplatform\n",
        "from google.cloud.aiplatform import hyperparameter_tuning as hpt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6246cc5582b5"
      },
      "source": [
        "### Initialize VertexAI SDK for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0debdc76f4ca"
      },
      "source": [
        "## Containerize training application code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6ac98289303"
      },
      "source": [
        "Before you can run a hyperparameter tuning job, you must create a source code file (training script) and a Dockerfile.\n",
        "\n",
        "The source code trains a model in the ML framework of your choice. In this example, you use TensorFlow to train a classification model.\n",
        "\n",
        "The Dockerfile includes all the commands needed to run the image. It installs all the libraries required by your training script, and set up the entry point for the training code. \n",
        "\n",
        "You create a directory for storing your source code file and Dockerfile."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b07b0aeb9309"
      },
      "outputs": [],
      "source": [
        "APPLICATION_DIR = \"hptune\"\n",
        "TRAINER_DIR = f\"{APPLICATION_DIR}/trainer\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bc7b72a9129a"
      },
      "outputs": [],
      "source": [
        "!mkdir -p $APPLICATION_DIR\n",
        "!mkdir -p $TRAINER_DIR"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e0914c763657"
      },
      "source": [
        "### Write the training script"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f0c3b8232eab"
      },
      "outputs": [],
      "source": [
        "%%writefile {TRAINER_DIR}/task.py\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "import argparse\n",
        "import hypertune\n",
        "\n",
        "NUM_EPOCHS = 10\n",
        "\n",
        "\n",
        "def get_args():\n",
        "  '''Parses args. Must include all hyperparameters you want to tune.'''\n",
        "\n",
        "  parser = argparse.ArgumentParser()\n",
        "  parser.add_argument(\n",
        "      '--learning_rate',\n",
        "      required=True,\n",
        "      type=float,\n",
        "      help='learning rate')\n",
        "  parser.add_argument(\n",
        "      '--momentum',\n",
        "      required=True,\n",
        "      type=float,\n",
        "      help='SGD momentum value')\n",
        "  parser.add_argument(\n",
        "      '--num_units',\n",
        "      required=True,\n",
        "      type=int,\n",
        "      help='number of units in last hidden layer')\n",
        "  args = parser.parse_args()\n",
        "  return args\n",
        "\n",
        "\n",
        "def preprocess_data(image, label):\n",
        "  '''Resizes and scales images.'''\n",
        "\n",
        "  image = tf.image.resize(image, (150,150))\n",
        "  return tf.cast(image, tf.float32) / 255., label\n",
        "\n",
        "\n",
        "def create_dataset():\n",
        "  '''Loads Horses Or Humans dataset and preprocesses data.'''\n",
        "\n",
        "  data, info = tfds.load(name='horses_or_humans', as_supervised=True, with_info=True)\n",
        "\n",
        "  # Create train dataset\n",
        "  train_data = data['train'].map(preprocess_data)\n",
        "  train_data  = train_data.shuffle(1000)\n",
        "  train_data  = train_data.batch(64)\n",
        "\n",
        "  # Create validation dataset\n",
        "  validation_data = data['test'].map(preprocess_data)\n",
        "  validation_data  = validation_data.batch(64)\n",
        "\n",
        "  return train_data, validation_data\n",
        "\n",
        "\n",
        "def create_model(num_units, learning_rate, momentum):\n",
        "  '''Defines and compiles model.'''\n",
        "\n",
        "  inputs = tf.keras.Input(shape=(150, 150, 3))\n",
        "  x = tf.keras.layers.Conv2D(16, (3, 3), activation='relu')(inputs)\n",
        "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
        "  x = tf.keras.layers.Conv2D(32, (3, 3), activation='relu')(x)\n",
        "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
        "  x = tf.keras.layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
        "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
        "  x = tf.keras.layers.Flatten()(x)\n",
        "  x = tf.keras.layers.Dense(num_units, activation='relu')(x)\n",
        "  outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n",
        "  model = tf.keras.Model(inputs, outputs)\n",
        "  model.compile(\n",
        "      loss='binary_crossentropy',\n",
        "      optimizer=tf.keras.optimizers.SGD(learning_rate=learning_rate, momentum=momentum),\n",
        "      metrics=['accuracy'])\n",
        "  return model\n",
        "\n",
        "\n",
        "def main():\n",
        "  args = get_args()\n",
        "  train_data, validation_data = create_dataset()\n",
        "  model = create_model(args.num_units, args.learning_rate, args.momentum)\n",
        "  history = model.fit(train_data, epochs=NUM_EPOCHS, validation_data=validation_data)\n",
        "\n",
        "  # DEFINE METRIC\n",
        "  hp_metric = history.history['val_accuracy'][-1]\n",
        "\n",
        "  hpt = hypertune.HyperTune()\n",
        "  hpt.report_hyperparameter_tuning_metric(\n",
        "      hyperparameter_metric_tag='accuracy',\n",
        "      metric_value=hp_metric,\n",
        "      global_step=NUM_EPOCHS)\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    main()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ff4babff0a8"
      },
      "source": [
        "### Understanding the training script\n",
        "\n",
        "Before you build the container, let's take a deeper look at the code. There are a few components that are specific to using the hyperparameter tuning service.\n",
        "\n",
        "The script imports the **hypertune library**:\n",
        "\n",
        "`import hypertune`\n",
        "\n",
        "The **function `get_args()`** defines a command-line argument for each hyperparameter that you want to tune. In this example, the hyperparameters tuned are the learning rate, the momentum value in the optimizer, and the number of units in the last hidden layer of the model. Feel free to experiment with others. The values passed in those arguments are then used to set the corresponding hyperparameter in the code.\n",
        "\n",
        "```\n",
        "def get_args():\n",
        "  '''Parses args. Must include all hyperparameters you want to tune.'''\n",
        "\n",
        "  parser = argparse.ArgumentParser()\n",
        "  parser.add_argument(\n",
        "      '--learning_rate',\n",
        "      required=True,\n",
        "      type=float,\n",
        "      help='learning rate')\n",
        "  parser.add_argument(\n",
        "      '--momentum',\n",
        "      required=True,\n",
        "      type=float,\n",
        "      help='SGD momentum value')\n",
        "  parser.add_argument(\n",
        "      '--num_units',\n",
        "      required=True,\n",
        "      type=int,\n",
        "      help='number of units in last hidden layer')\n",
        "  args = parser.parse_args()\n",
        "  return args\n",
        "```\n",
        "\n",
        "\n",
        "At the end of the `main()` function, the `hypertune` library is used to **define the metric you want to optimize**. In TensorFlow, the keras `model.fit` method returns a `History` object. The `History.history` attribute is a record of training loss values and metrics values at successive epochs. If you pass validation data to `model.fit` the `History.history` attribute includes validation loss and metrics values as well.\n",
        "\n",
        "```\n",
        "  hp_metric = history.history['val_accuracy'][-1]\n",
        "\n",
        "  hpt = hypertune.HyperTune()\n",
        "  hpt.report_hyperparameter_tuning_metric(\n",
        "      hyperparameter_metric_tag='accuracy',\n",
        "      metric_value=hp_metric,\n",
        "      global_step=NUM_EPOCHS)\n",
        " ```\n",
        " \n",
        "For example, if you trained a model for three epochs with validation data and provided accuracy as a metric, the `History.history` attribute would look similar to the following dictionary.\n",
        "\n",
        "```\n",
        "{\n",
        " \"accuracy\": [\n",
        "   0.7795261740684509,\n",
        "   0.9471358060836792,\n",
        "   0.9870933294296265\n",
        " ],\n",
        " \"loss\": [\n",
        "   0.6340447664260864,\n",
        "   0.16712145507335663,\n",
        "   0.04546636343002319\n",
        " ],\n",
        " \"val_accuracy\": [\n",
        "   0.3795261740684509,\n",
        "   0.4471358060836792,\n",
        "   0.4870933294296265\n",
        " ],\n",
        " \"val_loss\": [\n",
        "   2.044623374938965,\n",
        "   4.100203514099121,\n",
        "   3.0728273391723633\n",
        " ]\n",
        "```\n",
        "\n",
        "If you want the hyperparameter tuning service to discover the values that maximize the model's validation accuracy, you define the metric as the last entry (or `NUM_EPOCS` - 1) of the `val_accuracy` list. Then, pass this metric to an instance of `HyperTune`. You can pick whatever string you like for the `hyperparameter_metric_tag`, but you need to use the string again later when you kick off the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b74b64e9a3c5"
      },
      "source": [
        "### Write Dockerfile\n",
        "\n",
        "After writing your training code, you create a Dockerfile. In the Dockerfile, include all the commands needed to run the image. It installs all the necessary libraries, including the CloudML Hypertune library, and set up the entry point for the training code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ef3e973aed89"
      },
      "outputs": [],
      "source": [
        "%%writefile {APPLICATION_DIR}/Dockerfile\n",
        "\n",
        "FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-8\n",
        "\n",
        "WORKDIR /\n",
        "\n",
        "# Installs hypertune library\n",
        "RUN pip install cloudml-hypertune\n",
        "\n",
        "# Copies the trainer code to the Docker image.\n",
        "COPY trainer /trainer\n",
        "\n",
        "# Sets up the entry point to invoke the trainer.\n",
        "ENTRYPOINT [\"python\", \"-m\", \"trainer.task\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2ed53b6e259"
      },
      "source": [
        "### Enable Artifact Registry API\n",
        "\n",
        "First, you must enable the Artifact Registry API service for your project.\n",
        "\n",
        "Learn more about [Enabling service](https://cloud.google.com/artifact-registry/docs/enable-service)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "13970c971b51"
      },
      "outputs": [],
      "source": [
        "! gcloud services enable artifactregistry.googleapis.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "616150778c61"
      },
      "source": [
        "### Build the container\n",
        "\n",
        "You store the Docker image in Artifact Registry. First, create a Docker repository in Artifact Registry"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8d79730faf62"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "REPO_NAME='horses-app'\n",
        "\n",
        "if os.getenv(\"IS_TESTING\"):\n",
        "    ! sudo apt-get update --yes && sudo apt-get --only-upgrade --yes install google-cloud-sdk-cloud-run-proxy google-cloud-sdk-harbourbridge google-cloud-sdk-cbt google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-kpt google-cloud-sdk-local-extract google-cloud-sdk-minikube google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-python google-cloud-sdk-spanner-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-nomos google-cloud-sdk-package-go-module google-cloud-sdk-firestore-emulator kubectl google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-kubectl-oidc google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-grpc google-cloud-sdk-pubsub-emulator google-cloud-sdk-datalab google-cloud-sdk-skaffold google-cloud-sdk google-cloud-sdk-terraform-tools google-cloud-sdk-config-connector\n",
        "    ! gcloud components update --quiet\n",
        "\n",
        "!gcloud artifacts repositories create $REPO_NAME --repository-format=docker \\\n",
        "--location=$LOCATION --description=\"Docker repository\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94f77f953470"
      },
      "source": [
        "### Configure authentication to your private repo\n",
        "\n",
        "Before you push or pull container images, configure Docker to use the `gcloud` command-line tool to authenticate requests to `Artifact Registry` for your region."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3a47ce54c71d"
      },
      "outputs": [],
      "source": [
        "! gcloud auth configure-docker {LOCATION}-docker.pkg.dev --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fb8be3937a3"
      },
      "source": [
        "Define a variable with the URI of your Docker image in Artifact Registry:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "21f98973a812"
      },
      "outputs": [],
      "source": [
        "IMAGE_URI = (\n",
        "    f\"{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPO_NAME}/horse_human_hptune:latest\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "efd695af40f0"
      },
      "source": [
        "Then, build the container and push it to Artifact Registry by running the following:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "26be5eee7a08"
      },
      "outputs": [],
      "source": [
        "cd $APPLICATION_DIR"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ba5d83813d6"
      },
      "source": [
        "Push the image to the created artifact repository using Cloud-Build.\n",
        "\n",
        "**Note**: The following command automatically uses the Dockerfile from the directory it's being run from."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "74eddc8488f9"
      },
      "outputs": [],
      "source": [
        "! gcloud builds submit --region={LOCATION} --tag={IMAGE_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1b3dda0e2f68"
      },
      "source": [
        "## Configure a hyperparameter tuning job\n",
        "\n",
        "Now that your training application code is containerized, it's time to specify and run the hyperparameter tuning job.\n",
        "\n",
        "To launch the hyperparameter tuning job, you need to first define the `worker_pool_specs`, which specifies the machine type and Docker image. The following spec defines one `n1-standard-4` machine with one `NVIDIA Tesla T4` GPU as the accelerator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c066b586e18c"
      },
      "outputs": [],
      "source": [
        "# The spec of the worker pools including machine type and Docker image\n",
        "# Be sure to replace PROJECT_ID in the `image_uri` with your project.\n",
        "\n",
        "worker_pool_specs = [\n",
        "    {\n",
        "        \"machine_spec\": {\n",
        "            \"machine_type\": \"n1-standard-4\",\n",
        "            \"accelerator_type\": \"NVIDIA_TESLA_T4\",\n",
        "            \"accelerator_count\": 1,\n",
        "        },\n",
        "        \"replica_count\": 1,\n",
        "        \"container_spec\": {\n",
        "            \"image_uri\": f\"{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPO_NAME}/horse_human_hptune:latest\"\n",
        "        },\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58f02d71787a"
      },
      "source": [
        "### Define parameter spec\n",
        "\n",
        "Next, define the `parameter_spec`, which is a dictionary specifying the parameters you want to optimize. The **dictionary key** is the string you assigned to the command line argument for each hyperparameter, and the **dictionary value** is the parameter specification.\n",
        "\n",
        "For each hyperparameter, you need to define the `Type` as well as the bounds for the values that the tuning service tries. Hyperparameters can be of type `Double`, `Integer`, `Categorical`, or `Discrete`. If you select the type `Double` or `Integer`, you need to provide a minimum and maximum value. And if you select `Categorical` or `Discrete` you need to provide the values. For the `Double` and `Integer` types, you also need to provide the scaling value. Learn more about [Using an Appropriate Scale](https://www.youtube.com/watch?v=cSoK_6Rkbfg)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b129b91536a1"
      },
      "outputs": [],
      "source": [
        "# Dictionary representing parameters to optimize.\n",
        "# The dictionary key is the parameter_id, which is passed into your training\n",
        "# job as a command line argument,\n",
        "# And the dictionary value is the parameter specification of the metric.\n",
        "parameter_spec = {\n",
        "    \"learning_rate\": hpt.DoubleParameterSpec(min=0.001, max=1, scale=\"log\"),\n",
        "    \"momentum\": hpt.DoubleParameterSpec(min=0, max=1, scale=\"linear\"),\n",
        "    \"num_units\": hpt.DiscreteParameterSpec(values=[64, 128, 512], scale=None),\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5c37602d9bdf"
      },
      "source": [
        "The final spec to define is `metric_spec`, which is a dictionary representing the metric to optimize. The dictionary key is the `hyperparameter_metric_tag` that you set in your training application code, and the value is the optimization goal."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "abf4b7253354"
      },
      "outputs": [],
      "source": [
        "# Dictionary representing metrics to optimize.\n",
        "# The dictionary key is the metric_id, which is reported by your training job,\n",
        "# And the dictionary value is the optimization goal of the metric.\n",
        "metric_spec = {\"accuracy\": \"maximize\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3acfccaa053"
      },
      "source": [
        "Once the specs are defined, you create a `CustomJob`, which is the common spec that's used to run your job on each of the hyperparameter tuning trials."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2bc47496bcfd"
      },
      "outputs": [],
      "source": [
        "my_custom_job = aiplatform.CustomJob(\n",
        "    display_name=\"horses-humans-sdk-job\",\n",
        "    worker_pool_specs=worker_pool_specs,\n",
        "    staging_bucket=BUCKET_URI,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c82fd780f019"
      },
      "source": [
        "Then, create and run `HyperparameterTuningJob`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d7a1878e7872"
      },
      "outputs": [],
      "source": [
        "hp_job = aiplatform.HyperparameterTuningJob(\n",
        "    display_name=\"horses-humans-sdk-job\",\n",
        "    custom_job=my_custom_job,\n",
        "    metric_spec=metric_spec,\n",
        "    parameter_spec=parameter_spec,\n",
        "    max_trial_count=10,\n",
        "    parallel_trial_count=3,\n",
        ")\n",
        "\n",
        "hp_job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "956dc68681da"
      },
      "source": [
        "There are a few arguments to note:\n",
        "\n",
        "\n",
        "* **max_trial_count**: You need to put an upper bound on the number of trials the service runs. More trials generally leads to better results, but there is a point of diminishing returns, after which additional trials have little or no effect on the metric you're trying to optimize. The best practice is to start with a smaller number of trials and get a sense of how impactful your chosen hyperparameters are before scaling up.\n",
        "\n",
        "* **parallel_trial_count**: If you use parallel trials, the service provisions multiple training processing clusters. Increasing the number of parallel trials reduces the amount of time the hyperparameter tuning job takes to run; however, it can reduce the effectiveness of the job overall. This is because the default tuning strategy uses results of previous trials to inform the assignment of values in subsequent trials.\n",
        "\n",
        "* **search_algorithm**: You can set the search algorithm to grid, random, or default (None). If you do not specify an algorithm, as shown in this example, the default option applies Bayesian optimization to search the space of possible hyperparameter values and is the recommended algorithm. Learn more about [Hyperparameter tuning using Bayesian Optimization](https://cloud.google.com/blog/products/ai-machine-learning/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afd332595672"
      },
      "source": [
        "## Examine results\n",
        "\n",
        "Click on the generated link in the output to see your run in the Cloud Console. When the job completes, you see the results of the tuning trials.\n",
        "\n",
        "You can sort the results by the optimization metric and then set the hyperparamers in your training application code to the values from the trial with the highest accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b641b26f883d"
      },
      "source": [
        "![console_ui_results](tuning_results.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# Delete artifact registry repo\n",
        "! gcloud artifacts repositories delete $REPO_NAME --location $LOCATION --quiet\n",
        "\n",
        "delete_custom_job = True\n",
        "delete_application_directory = True\n",
        "delete_bucket = False\n",
        "\n",
        "# Delete hptune job\n",
        "if delete_custom_job:\n",
        "    try:\n",
        "        hp_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "# Delete application directory\n",
        "if delete_application_directory:\n",
        "    !cd .. && rm -rf $APPLICATION_DIR\n",
        "\n",
        "# Delete bucket\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "hyperparameter_tuning_tensorflow.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}