notebooks/official/migration/UJ11 Vertex SDK Hyperparameter Tuning.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:migration,new"
      },
      "source": [
        "# Vertex AI: Vertex AI Migration: Hyperparameter Tuning\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/vertex-ai-samples/tree/master/notebooks/official/migration/UJ11%20Vertex%20SDK%20Hyperparameter%20Tuning.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/vertex-ai-samples/tree/master/notebooks/official/migration/UJ11%20Vertex%20SDK%20Hyperparameter%20Tuning.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:custom,boston,lrg"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Boston Housing Prices dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html). The version of the dataset you will use in this tutorial is built into TensorFlow. The trained model predicts the median price of a house in units of 1K USD."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_local"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "If you are using Colab or Google Cloud Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
        "\n",
        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
        "\n",
        "- The Cloud Storage SDK\n",
        "- Git\n",
        "- Python 3\n",
        "- virtualenv\n",
        "- Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
        "\n",
        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
        "\n",
        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
        "\n",
        "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.  Activate the virtual environment.\n",
        "\n",
        "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
        "\n",
        "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
        "\n",
        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the latest version of Vertex SDK for Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Google Cloud Notebook\n",
        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    USER_FLAG = \"--user\"\n",
        "else:\n",
        "    USER_FLAG = \"\"\n",
        "\n",
        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_storage"
      },
      "source": [
        "Install the latest GA version of *google-cloud-storage* library as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_storage"
      },
      "outputs": [],
      "source": [
        "! pip3 install -U google-cloud-storage $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "restart"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### GPU runtime\n",
        "\n",
        "This tutorial does not require a GPU runtime.\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
        "\n",
        "4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
        "\n",
        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_project_id"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID:\", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_gcloud_project_id"
      },
      "outputs": [],
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-east1`\n",
        "\n",
        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
        "\n",
        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
        "\n",
        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
        "\n",
        "**Click Create service account**.\n",
        "\n",
        "In the **Service account name** field, enter a name, and click **Create**.\n",
        "\n",
        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
        "\n",
        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gcp_authenticate"
      },
      "outputs": [],
      "source": [
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "import os\n",
        "import sys\n",
        "\n",
        "# If on Google Cloud Notebook, then don't execute this code\n",
        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_bucket"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex SDK for Python\n",
        "\n",
        "Initialize the Vertex SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region\n",
        "\n",
        "*Note*: TF releases before 2.3 for GPU support will fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
        "    TRAIN_GPU, TRAIN_NGPU = (\n",
        "        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
        "        int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
        "    )\n",
        "else:\n",
        "    TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
        "\n",
        "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
        "    DEPLOY_GPU, DEPLOY_NGPU = (\n",
        "        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
        "        int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n",
        "    )\n",
        "else:\n",
        "    DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for training and prediction.\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"IS_TESTING_TF\"):\n",
        "    TF = os.getenv(\"IS_TESTING_TF\")\n",
        "else:\n",
        "    TF = \"2-1\"\n",
        "\n",
        "if TF[0] == \"2\":\n",
        "    if TRAIN_GPU:\n",
        "        TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "    if DEPLOY_GPU:\n",
        "        DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "else:\n",
        "    if TRAIN_GPU:\n",
        "        TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "    if DEPLOY_GPU:\n",
        "        DEPLOY_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "\n",
        "TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n",
        "DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n",
        "\n",
        "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training,prediction"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for training and prediction.\n",
        "\n",
        "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you will use for for training and prediction.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "*Note: The following is not supported for training:*\n",
        "\n",
        " - `standard`: 2 vCPUs\n",
        " - `highcpu`: 2, 4 and 8 vCPUs\n",
        "\n",
        "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training,prediction"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n",
        "    MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n",
        "else:\n",
        "    MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", TRAIN_COMPUTE)\n",
        "\n",
        "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n",
        "    MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n",
        "else:\n",
        "    MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "examine_training_package"
      },
      "source": [
        "### Examine the training package\n",
        "\n",
        "#### Package layout\n",
        "\n",
        "Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n",
        "\n",
        "- PKG-INFO\n",
        "- README.md\n",
        "- setup.cfg\n",
        "- setup.py\n",
        "- trainer\n",
        "  - \\_\\_init\\_\\_.py\n",
        "  - task.py\n",
        "\n",
        "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n",
        "\n",
        "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*, when we referred to it in the worker pool specification, we replace the directory slash with a dot (`trainer.task`) and dropped the file suffix (`.py`).\n",
        "\n",
        "#### Package Assembly\n",
        "\n",
        "In the following cells, you will assemble the training package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "examine_training_package"
      },
      "outputs": [],
      "source": [
        "# Make folder for Python training script\n",
        "! rm -rf custom\n",
        "! mkdir custom\n",
        "\n",
        "# Add package information\n",
        "! touch custom/README.md\n",
        "\n",
        "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n",
        "! echo \"$setup_cfg\" > custom/setup.cfg\n",
        "\n",
        "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n    install_requires=[\\n\\n        'tensorflow_datasets==1.3.0',\\n\\n    ],\\n\\n    packages=setuptools.find_packages())\"\n",
        "! echo \"$setup_py\" > custom/setup.py\n",
        "\n",
        "pkg_info = \"Metadata-Version: 1.0\\n\\nName: Boston Housing tabular regression\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n",
        "! echo \"$pkg_info\" > custom/PKG-INFO\n",
        "\n",
        "# Make the training subfolder\n",
        "! mkdir custom/trainer\n",
        "! touch custom/trainer/__init__.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "taskpy_contents:hpt,boston"
      },
      "source": [
        "#### Task.py contents\n",
        "\n",
        "In the next cell, you write the contents of the hyperparameter tuning script task.py. I won't go into detail, it's just there for you to browse. In summary:\n",
        "\n",
        "- Parse the command line arguments for the hyperparameter settings for the current trial.\n",
        " - Get the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n",
        "- Download and preprocess the Boston Housing dataset.\n",
        "- Build a DNN model.\n",
        "- The number of units per dense layer and learning rate hyperparameter values are used during the build and compile of the model.\n",
        "- A definition of a callback `HPTCallback` which obtains the validation loss at the end of each epoch (`on_epoch_end()`) and reports it to the hyperparameter tuning service using `hpt.report_hyperparameter_tuning_metric()`.\n",
        "- Train the model with the `fit()` method and specify a callback which will report the validation loss back to the hyperparameter tuning service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "taskpy_contents:hpt,boston"
      },
      "outputs": [],
      "source": [
        "%%writefile custom/trainer/task.py\n",
        "# Custom Training for Boston Housing\n",
        "\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.client import device_lib\n",
        "from hypertune import HyperTune\n",
        "import numpy as np\n",
        "import argparse\n",
        "import os\n",
        "import sys\n",
        "tfds.disable_progress_bar()\n",
        "\n",
        "parser = argparse.ArgumentParser()\n",
        "parser.add_argument('--model-dir', dest='model_dir',\n",
        "                    default=os.getenv('AIP_MODEL_DIR'), type=str, help='Model dir.')\n",
        "parser.add_argument('--lr', dest='lr',\n",
        "                    default=0.001, type=float,\n",
        "                    help='Learning rate.')\n",
        "parser.add_argument('--decay', dest='decay',\n",
        "                    default=0.98, type=float,\n",
        "                    help='Decay rate')\n",
        "parser.add_argument('--units', dest='units',\n",
        "                    default=64, type=int,\n",
        "                    help='Number of units.')\n",
        "parser.add_argument('--epochs', dest='epochs',\n",
        "                    default=20, type=int,\n",
        "                    help='Number of epochs.')\n",
        "parser.add_argument('--steps', dest='steps',\n",
        "                    default=200, type=int,\n",
        "                    help='Number of steps per epoch.')\n",
        "parser.add_argument('--param-file', dest='param_file',\n",
        "                    default='/tmp/param.txt', type=str,\n",
        "                    help='Output file for parameters')\n",
        "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n",
        "                    help='distributed training strategy')\n",
        "args = parser.parse_args()\n",
        "\n",
        "print('Python Version = {}'.format(sys.version))\n",
        "print('TensorFlow Version = {}'.format(tf.__version__))\n",
        "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n",
        "\n",
        "\n",
        "def make_dataset():\n",
        "\n",
        "  # Scaling Boston Housing data features\n",
        "  def scale(feature):\n",
        "    max = np.max(feature)\n",
        "    feature = (feature / max).astype(np.float)\n",
        "    return feature, max\n",
        "\n",
        "  (x_train, y_train), (x_test, y_test) = tf.keras.datasets.boston_housing.load_data(\n",
        "    path=\"boston_housing.npz\", test_split=0.2, seed=113\n",
        "  )\n",
        "  params = []\n",
        "  for _ in range(13):\n",
        "    x_train[_], max = scale(x_train[_])\n",
        "    x_test[_], _ = scale(x_test[_])\n",
        "    params.append(max)\n",
        "\n",
        "  # store the normalization (max) value for each feature\n",
        "  with tf.io.gfile.GFile(args.param_file, 'w') as f:\n",
        "    f.write(str(params))\n",
        "  return (x_train, y_train), (x_test, y_test)\n",
        "\n",
        "# Build the Keras model\n",
        "def build_and_compile_dnn_model():\n",
        "  model = tf.keras.Sequential([\n",
        "      tf.keras.layers.Dense(args.units, activation='relu', input_shape=(13,)),\n",
        "      tf.keras.layers.Dense(args.units, activation='relu'),\n",
        "      tf.keras.layers.Dense(1, activation='linear')\n",
        "  ])\n",
        "  model.compile(\n",
        "      loss='mse',\n",
        "      optimizer=tf.keras.optimizers.RMSprop(learning_rate=args.lr, decay=args.decay))\n",
        "  return model\n",
        "\n",
        "\n",
        "model = build_and_compile_dnn_model()\n",
        "\n",
        "# Instantiate the HyperTune reporting object\n",
        "hpt = HyperTune()\n",
        "\n",
        "# Reporting callback\n",
        "class HPTCallback(tf.keras.callbacks.Callback):\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        global hpt\n",
        "        hpt.report_hyperparameter_tuning_metric(\n",
        "        hyperparameter_metric_tag='val_loss',\n",
        "        metric_value=logs['val_loss'],\n",
        "        global_step=epoch)\n",
        "\n",
        "# Train the model\n",
        "BATCH_SIZE = 16\n",
        "(x_train, y_train), (x_test, y_test) = make_dataset()\n",
        "model.fit(x_train, y_train, epochs=args.epochs, batch_size=BATCH_SIZE, validation_split=0.1, callbacks=[HPTCallback()])\n",
        "model.save(args.model_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tarball_training_script"
      },
      "source": [
        "#### Store training script on your Cloud Storage bucket\n",
        "\n",
        "Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tarball_training_script"
      },
      "outputs": [],
      "source": [
        "! rm -f custom.tar custom.tar.gz\n",
        "! tar cvf custom.tar custom\n",
        "! gzip custom.tar\n",
        "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_a_model:migration"
      },
      "source": [
        "## Train a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hyperparameter_tuning:migration,new,mbsdk"
      },
      "source": [
        "### [training.using-hyperparamter-tuning](https://cloud.google.com/vertex-ai/docs/training/using-hyperparameter-tuning)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_machine_specification"
      },
      "source": [
        "### Prepare your machine specification\n",
        "\n",
        "Now define the machine specification for your custom training job. This tells Vertex what type of machine instance to provision for the training.\n",
        "  - `machine_type`: The type of GCP instance to provision -- e.g., n1-standard-8.\n",
        "  - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.\n",
        "  - `accelerator_count`: The number of accelerators."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_machine_specification"
      },
      "outputs": [],
      "source": [
        "if TRAIN_GPU:\n",
        "    machine_spec = {\n",
        "        \"machine_type\": TRAIN_COMPUTE,\n",
        "        \"accelerator_type\": TRAIN_GPU,\n",
        "        \"accelerator_count\": TRAIN_NGPU,\n",
        "    }\n",
        "else:\n",
        "    machine_spec = {\"machine_type\": TRAIN_COMPUTE, \"accelerator_count\": 0}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_disk_specification"
      },
      "source": [
        "### Prepare your disk specification\n",
        "\n",
        "(optional) Now define the disk specification for your custom training job. This tells Vertex what type and size of disk to provision in each machine instance for the training.\n",
        "\n",
        "  - `boot_disk_type`: Either SSD or Standard. SSD is faster, and Standard is less expensive. Defaults to SSD.\n",
        "  - `boot_disk_size_gb`: Size of disk in GB."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_disk_specification"
      },
      "outputs": [],
      "source": [
        "DISK_TYPE = \"pd-ssd\"  # [ pd-ssd, pd-standard]\n",
        "DISK_SIZE = 200  # GB\n",
        "\n",
        "disk_spec = {\"boot_disk_type\": DISK_TYPE, \"boot_disk_size_gb\": DISK_SIZE}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_worker_pool_specification:prebuilt_container"
      },
      "source": [
        "### Define the worker pool specification\n",
        "\n",
        "Next, you define the worker pool specification for your custom training job. The worker pool specification will consist of the following:\n",
        "\n",
        "- `replica_count`: The number of instances to provision of this machine type.\n",
        "- `machine_spec`: The hardware specification.\n",
        "- `disk_spec` : (optional) The disk storage specification.\n",
        "\n",
        "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n",
        "\n",
        "Let's dive deeper now into the python package specification:\n",
        "\n",
        "-`executor_image_spec`: This is the docker image which is configured for your custom training job.\n",
        "\n",
        "-`package_uris`: This is a list of the locations (URIs) of your python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual python files or a zip (archive) of an entire package. In the later case, the job service will unzip (unarchive) the contents into the docker image.\n",
        "\n",
        "-`python_module`: The Python module (script) to invoke for running the custom training job. In this example, you will be invoking `trainer.task.py` -- note that it was not neccessary to append the `.py` suffix.\n",
        "\n",
        "-`args`: The command line arguments to pass to the corresponding Pythom module. In this example, you will be setting:\n",
        "  - `\"--model-dir=\" + MODEL_DIR` : The Cloud Storage location where to store the model artifacts. There are two ways to tell the training script where to save the model artifacts:\n",
        "      - direct: You pass the Cloud Storage location as a command line argument to your training script (set variable `DIRECT = True`), or\n",
        "      - indirect: The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script (set variable `DIRECT = False`). In this case, you tell the service the model artifact location in the job specification.\n",
        "  - `\"--epochs=\" + EPOCHS`: The number of epochs for training.\n",
        "  - `\"--steps=\" + STEPS`: The number of steps (batches) per epoch.\n",
        "  - `\"--distribute=\" + TRAIN_STRATEGY\"` : The training distribution strategy to use for single or distributed training.\n",
        "     - `\"single\"`: single device.\n",
        "     - `\"mirror\"`: all GPU devices on a single compute instance.\n",
        "     - `\"multi\"`: all GPU devices on all compute instances."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_worker_pool_specification:prebuilt_container"
      },
      "outputs": [],
      "source": [
        "JOB_NAME = \"custom_job_\" + TIMESTAMP\n",
        "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, JOB_NAME)\n",
        "\n",
        "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n",
        "    TRAIN_STRATEGY = \"single\"\n",
        "else:\n",
        "    TRAIN_STRATEGY = \"mirror\"\n",
        "\n",
        "EPOCHS = 20\n",
        "STEPS = 100\n",
        "\n",
        "DIRECT = True\n",
        "if DIRECT:\n",
        "    CMDARGS = [\n",
        "        \"--model-dir=\" + MODEL_DIR,\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "        \"--distribute=\" + TRAIN_STRATEGY,\n",
        "    ]\n",
        "else:\n",
        "    CMDARGS = [\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "        \"--distribute=\" + TRAIN_STRATEGY,\n",
        "    ]\n",
        "\n",
        "worker_pool_spec = [\n",
        "    {\n",
        "        \"replica_count\": 1,\n",
        "        \"machine_spec\": machine_spec,\n",
        "        \"disk_spec\": disk_spec,\n",
        "        \"python_package_spec\": {\n",
        "            \"executor_image_uri\": TRAIN_IMAGE,\n",
        "            \"package_uris\": [BUCKET_NAME + \"/trainer_boston.tar.gz\"],\n",
        "            \"python_module\": \"trainer.task\",\n",
        "            \"args\": CMDARGS,\n",
        "        },\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_job:migration,new,mbsdk"
      },
      "source": [
        "### [training.create-custom-job](https://cloud.google.com/vertex-ai/docs/training/create-custom-job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "source": [
        "## Create a custom job\n",
        "\n",
        "Use the class `CustomJob` to create a custom job, such as for hyperparameter tuning, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `worker_pool_specs`: The specification for the corresponding VM instances."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "job = aip.CustomJob(\n",
        "    display_name=\"boston_\" + TIMESTAMP, worker_pool_specs=worker_pool_spec\n",
        ")\n",
        "\n",
        "# print(job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_hpt_job:mbsdk"
      },
      "source": [
        "## Create a hyperparameter tuning job\n",
        "\n",
        "Use the class `HyperparameterTuningJob` to create a hyperparameter tuning job, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `custom_job`: The worker pool spec from this custom job applies to the CustomJobs created in all the trials.\n",
        "- `metrics_spec`: The metrics to optimize. The dictionary key is the metric_id, which is reported by your training job, and the dictionary value is the optimization goal of the metric('minimize' or 'maximize').\n",
        "- `parameter_spec`: The parameters to optimize. The dictionary key is the metric_id, which is passed into your training job as a command line key word argument, and the dictionary value is the parameter specification of the metric."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_hpt_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "from google.cloud.aiplatform import hyperparameter_tuning as hpt\n",
        "\n",
        "hpt_job = aip.HyperparameterTuningJob(\n",
        "    display_name=\"boston_\" + TIMESTAMP,\n",
        "    custom_job=job,\n",
        "    metric_spec={\n",
        "        \"val_loss\": \"minimize\",\n",
        "    },\n",
        "    parameter_spec={\n",
        "        \"lr\": hpt.DoubleParameterSpec(min=0.001, max=0.1, scale=\"log\"),\n",
        "        \"units\": hpt.IntegerParameterSpec(min=4, max=128, scale=\"linear\"),\n",
        "    },\n",
        "    max_trial_count=6,\n",
        "    parallel_trial_count=1,\n",
        ")\n",
        "\n",
        "# print(hpt_job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "source": [
        "## Run the hyperparameter tuning job\n",
        "\n",
        "Use the `run()` method to execute the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "hpt_job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    INFO:google.cloud.aiplatform.jobs:Creating HyperparameterTuningJob\n",
        "    INFO:google.cloud.aiplatform.jobs:HyperparameterTuningJob created. Resource name: projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048\n",
        "    INFO:google.cloud.aiplatform.jobs:To use this HyperparameterTuningJob in another session:\n",
        "    INFO:google.cloud.aiplatform.jobs:hpt_job = aiplatform.HyperparameterTuningJob.get('projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048')\n",
        "    INFO:google.cloud.aiplatform.jobs:View HyperparameterTuningJob:\n",
        "    https://console.cloud.google.com/ai/platform/locations/us-central1/training/760969798560514048?project=759209241365\n",
        "    INFO:google.cloud.aiplatform.jobs:HyperparameterTuningJob projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:HyperparameterTuningJob projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    ...\n",
        "    INFO:google.cloud.aiplatform.jobs:HyperparameterTuningJob projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048 current state:\n",
        "    JobState.JOB_STATE_SUCCEEDED\n",
        "    INFO:google.cloud.aiplatform.jobs:HyperparameterTuningJob run completed. Resource name: projects/759209241365/locations/us-central1/hyperparameterTuningJobs/760969798560514048"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "source": [
        "### Display the hyperparameter tuning job trial results\n",
        "\n",
        "After the hyperparameter tuning job has completed, the property `trials` will return the results for each trial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "outputs": [],
      "source": [
        "print(hpt_job.trials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    [id: \"1\"\n",
        "    state: SUCCEEDED\n",
        "    parameters {\n",
        "      parameter_id: \"lr\"\n",
        "      value {\n",
        "        number_value: 0.010000000000000002\n",
        "      }\n",
        "    }\n",
        "    parameters {\n",
        "      parameter_id: \"units\"\n",
        "      value {\n",
        "        number_value: 66.0\n",
        "      }\n",
        "    }\n",
        "    final_measurement {\n",
        "      step_count: 19\n",
        "      metrics {\n",
        "        metric_id: \"val_loss\"\n",
        "        value: 24.61802691948123\n",
        "      }\n",
        "    }\n",
        "    start_time {\n",
        "      seconds: 1629414344\n",
        "      nanos: 232151761\n",
        "    }\n",
        "    end_time {\n",
        "      seconds: 1629414825\n",
        "    }\n",
        "    , id: \"2\"\n",
        "    state: SUCCEEDED\n",
        "    parameters {\n",
        "      parameter_id: \"lr\"\n",
        "      value {\n",
        "        number_value: 0.0036327633937958217\n",
        "      }\n",
        "    }\n",
        "    parameters {\n",
        "      parameter_id: \"units\"\n",
        "      value {\n",
        "        number_value: 94.0\n",
        "      }\n",
        "    }\n",
        "    final_measurement {\n",
        "      step_count: 19\n",
        "      metrics {\n",
        "        metric_id: \"val_loss\"\n",
        "        value: 25.499705989186356\n",
        "      }\n",
        "    }\n",
        "    start_time {\n",
        "      seconds: 1629414960\n",
        "      nanos: 618333580\n",
        "    }\n",
        "    end_time {\n",
        "      seconds: 1629415021\n",
        "    }\n",
        "    ...\n",
        "    , id: \"6\"\n",
        "    state: SUCCEEDED\n",
        "    parameters {\n",
        "      parameter_id: \"lr\"\n",
        "      value {\n",
        "        number_value: 0.002775175422799751\n",
        "      }\n",
        "    }\n",
        "    parameters {\n",
        "      parameter_id: \"units\"\n",
        "      value {\n",
        "        number_value: 104.0\n",
        "      }\n",
        "    }\n",
        "    final_measurement {\n",
        "      step_count: 7\n",
        "      metrics {\n",
        "        metric_id: \"val_loss\"\n",
        "        value: 24.655450123112377\n",
        "      }\n",
        "    }\n",
        "    start_time {\n",
        "      seconds: 1629415735\n",
        "      nanos: 998711852\n",
        "    }\n",
        "    end_time {\n",
        "      seconds: 1629415766\n",
        "    }\n",
        "    ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "- Dataset\n",
        "- Pipeline\n",
        "- Model\n",
        "- Endpoint\n",
        "- AutoML Training Job\n",
        "- Batch Job\n",
        "- Custom Job\n",
        "- Hyperparameter Tuning Job\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "delete_all = True\n",
        "\n",
        "if delete_all:\n",
        "    # Delete the dataset using the Vertex dataset object\n",
        "    try:\n",
        "        if \"dataset\" in globals():\n",
        "            dataset.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the model using the Vertex model object\n",
        "    try:\n",
        "        if \"model\" in globals():\n",
        "            model.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the endpoint using the Vertex endpoint object\n",
        "    try:\n",
        "        if \"endpoint\" in globals():\n",
        "            endpoint.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the AutoML or Pipeline trainig job\n",
        "    try:\n",
        "        if \"dag\" in globals():\n",
        "            dag.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the custom trainig job\n",
        "    try:\n",
        "        if \"job\" in globals():\n",
        "            job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
        "    try:\n",
        "        if \"batch_predict_job\" in globals():\n",
        "            batch_predict_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
        "    try:\n",
        "        if \"hpt_job\" in globals():\n",
        "            hpt_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    if \"BUCKET_NAME\" in globals():\n",
        "        ! gsutil rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "UJ11 Vertex SDK Hyperparameter Tuning.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}