notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Vertex AI Experiments: Custom training autologging - Local script\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24743cf4a1e1"
      },
      "source": [
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "As a Data Scientist experimenting with large models, you need a way to run experiments on a scalable training service to log parameters and metrics. This guarantees reproducibility.\n",
        "\n",
        "With Vertex AI training and Vertex AI Experiments autologging integration, you can run your ML experiments at scale and autolog their parameters and metrics\n",
        "with the `enable_autolog` argument.\n",
        "\n",
        "Learn more about [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments)\n",
        "and how to [Autolog data to an experiment run](https://cloud.google.com/vertex-ai/docs/experiments/autolog-data)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to autolog paramenters and metrics of an ML experiment running on Vertex AI Training by leveraging the integration with Vertex AI Experiments.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Experiments\n",
        "- Vertex AI Training\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Formalize model experiment in a script\n",
        "- Run model traning using local script on Vertex AI Training\n",
        "- Check out ML experiment parameters and metrics in Vertex AI Experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The [Glass Identification dataset](https://archive-beta.ics.uci.edu/dataset/42/glass+identification) is a dataset from USA Forensic Science Service with 6 types of glass defined in terms of their oxide content (for example, Na, Fe, K). The goal is to classify the types of glass based on oxide features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "USER = \"\"\n",
        "! pip3 install {USER} --upgrade google-cloud-aiplatform --quiet --no-warn-conflicts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1s_lfsWxhctH"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-_8nZXd7NqIj"
      },
      "source": [
        "### UUID\n",
        "If you’re in a live tutorial session, you may be using a shared test account or project. To avoid name collisions between users on resources created, create a Universal Unique Identifier (uuid) for each instance session. Append the UUID to the name of the resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dY-WpyyzNtS0"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of length 8\n",
        "def generate_uuid():\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=8))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74ccc9e52986"
      },
      "source": [
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de775a3773ba"
      },
      "source": [
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "254614fa0c46"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef21552ccea8"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "603adbbf0532"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6b2ccc891ed"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Set Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below.\n",
        "\n",
        "*Note:* The code for automatically finding your service account works on a user-managed Workbench AI noteboook.\n",
        "If you are using a fully-managed notebook, you'll need to manually enter your service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M4WZi4CDJ39n"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "IS_COLAB = False\n",
        "\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    if IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Training\n",
        "\n",
        "Run the following commands to grant your service account access to read and update metadata in Vertex AI ML Metadata while the custom training job is running -- you only need to run these once per service account. Check out the [documentation](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training#create_a_service_account_with_required_permissions) to get more information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mI3IJONMJ39n"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UlICvTdOtyLN"
      },
      "source": [
        "### Set up project template\n",
        "Set the folder you use in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rxRaeiQIt6kn"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "TUTORIAL_DIR = os.path.join(\n",
        "    os.getcwd(), \"custom_training_autologging_local_script_tutorial\"\n",
        ")\n",
        "os.makedirs(TUTORIAL_DIR, exist_ok=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-flLpWUeOX_n"
      },
      "source": [
        "### Get dataset\n",
        "Get the glass classification dataset from the public Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eRBPBlj1PBUK"
      },
      "outputs": [],
      "source": [
        "SOURCE_DATA_URL = \"gs://cloud-samples-data/vertex-ai/dataset-management/datasets/uci_glass_preprocessed/glass.csv\"\n",
        "DESTINATION_DATA_URL = f\"{BUCKET_URI}/data/glass.csv\"\n",
        "\n",
        "! gsutil cp $SOURCE_DATA_URL $DESTINATION_DATA_URL"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "from google.cloud import aiplatform as vertex_ai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rLzhLCJVKIHO"
      },
      "source": [
        "### Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sWa5mNJEKKEx"
      },
      "outputs": [],
      "source": [
        "# Training\n",
        "EXPERIMENT_NAME = f\"glass-classification-{UUID}\"\n",
        "TRAIN_SCRIPT_PATH = os.path.join(TUTORIAL_DIR, \"task.py\")\n",
        "JOB_DISPLAY_NAME = f\"sklearn-autologged-custom-job-{UUID}\"\n",
        "PRE_BUILT_TRAINING_CONTAINER_IMAGE_URI = (\n",
        "    f\"{REGION.split('-')[0]}-docker.pkg.dev/vertex-ai/training/tf-cpu.2-12.py310:latest\"\n",
        ")\n",
        "MODEL_FILE_URI = f\"{BUCKET_URI}/models/model.joblib\"\n",
        "DESTINATION_DATA_PATH = DESTINATION_DATA_URL.replace(\"gs://\", \"/gcs/\")\n",
        "MODEL_FILE_PATH = MODEL_FILE_URI.replace(\"gs://\", \"/gcs/\")\n",
        "REPLICA_COUNT = 1\n",
        "TRAIN_MACHINE_TYPE = \"n1-standard-4\"\n",
        "TRAINING_JOBS_URI = f\"{BUCKET_URI}/jobs\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kyr59QdyhctK"
      },
      "outputs": [],
      "source": [
        "vertex_ai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJoFEV5VNWb3"
      },
      "source": [
        "### Create an experiment for tracking training parameters and metrics\n",
        "\n",
        "To start, initiate an experiment using the `init()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8ZEB_90wNc2J"
      },
      "outputs": [],
      "source": [
        "vertex_ai.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=REGION,\n",
        "    staging_bucket=BUCKET_URI,\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig4h3BP_EuqF"
      },
      "source": [
        "### Train a scikit-learn model with a prebuilt container\n",
        "\n",
        "Then, you train a custom model using a prebuilt container for scikit-learn models.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeKHxs4TK0Es"
      },
      "source": [
        "#### Create scikit-learn training script"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zOKUe_J3JWBs"
      },
      "outputs": [],
      "source": [
        "task_script = f\"\"\"\n",
        "#!/usr/bin/env python3\n",
        "\n",
        "'''\n",
        "A simple module to train a classifier on the glass dataset.\n",
        "'''\n",
        "\n",
        "# Libraries\n",
        "import argparse\n",
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "import joblib\n",
        "\n",
        "\n",
        "# Variables\n",
        "DATA_PATH = '{DESTINATION_DATA_PATH}'\n",
        "MODEL_PATH = '{MODEL_FILE_PATH}'\n",
        "TEST_SIZE = 0.2\n",
        "SEED = 8\n",
        "\n",
        "# Helpers\n",
        "def read_data(path):\n",
        "    df = pd.read_csv(path)\n",
        "    return df\n",
        "\n",
        "\n",
        "def split_data(df):\n",
        "    y = df.pop('glass_type')\n",
        "    X = df\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=TEST_SIZE, random_state=SEED)\n",
        "    return X_train, X_test, y_train, y_test\n",
        "\n",
        "\n",
        "def train_model(X_train, y_train):\n",
        "    model = RandomForestClassifier(n_estimators=5)\n",
        "    model.fit(X_train, y_train)\n",
        "    return model\n",
        "\n",
        "\n",
        "def evaluate_model(model, X_test, y_test):\n",
        "    y_pred = model.predict(X_test)\n",
        "    accuracy = accuracy_score(y_test, y_pred)\n",
        "    return accuracy\n",
        "\n",
        "\n",
        "def save_model(model, path):\n",
        "    p = Path(path)\n",
        "    if not p.parent.exists():\n",
        "      p.parent.mkdir(parents=True)\n",
        "    joblib.dump(model, path)\n",
        "\n",
        "\n",
        "def main():\n",
        "\n",
        "    # Read data\n",
        "    df = read_data(DATA_PATH)\n",
        "\n",
        "    # Split data\n",
        "    X_train, X_test, y_train, y_test = split_data(df)\n",
        "\n",
        "    # Train model\n",
        "    model = train_model(X_train, y_train)\n",
        "\n",
        "    # Evaluate model\n",
        "    accuracy = evaluate_model(model, X_test, y_test)\n",
        "    print('Model accuracy:', accuracy)\n",
        "\n",
        "    # Save model\n",
        "    save_model(model, MODEL_PATH)\n",
        "\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "\n",
        "    # Run main\n",
        "    main()\n",
        "\"\"\"\n",
        "\n",
        "with open(TRAIN_SCRIPT_PATH, \"w\") as train_file:\n",
        "    train_file.write(task_script)\n",
        "train_file.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tXILrH3Kr4p"
      },
      "source": [
        "#### Define custom training job\n",
        "\n",
        "Define a custom job with the prebuilt container image for training code packaged as Python script. In this case, you set `enable_autolog=True` to automatically track parameters and metrics after the training job completes. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jACJpf78LFoy"
      },
      "outputs": [],
      "source": [
        "job = vertex_ai.CustomJob.from_local_script(\n",
        "    project=PROJECT_ID,\n",
        "    staging_bucket=TRAINING_JOBS_URI,\n",
        "    display_name=JOB_DISPLAY_NAME,\n",
        "    script_path=TRAIN_SCRIPT_PATH,\n",
        "    container_uri=PRE_BUILT_TRAINING_CONTAINER_IMAGE_URI,\n",
        "    requirements=[\"pandas\", \"scikit-learn\"],\n",
        "    replica_count=REPLICA_COUNT,\n",
        "    machine_type=TRAIN_MACHINE_TYPE,\n",
        "    enable_autolog=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xfaE5rK-MYxO"
      },
      "source": [
        "### Run custom training job\n",
        "Next, you run the training job using the method `run`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4-Jn9rJeMUZa"
      },
      "outputs": [],
      "source": [
        "job.run(experiment=EXPERIMENT_NAME, service_account=SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYwi0B3cd515"
      },
      "source": [
        "### Get your autologged experiment\n",
        "\n",
        "After you train your model, you can get parameters and metrics of the autologged experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IQCXPih6d9nA"
      },
      "outputs": [],
      "source": [
        "experiment_df = vertex_ai.get_experiment_df(experiment=EXPERIMENT_NAME)\n",
        "experiment_df.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ffcedb5809e4"
      },
      "source": [
        "Also you can get custom training job metadata associated with the experiment you run. You resume the logged experiments and use `get_logged_custom_jobs()` to get all `CustomJobs` resources associated to this experiment run. Then you use `job_spec` to print custom job metadata such as the training python package, training resources and more. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fI2xd4DbJvHC"
      },
      "outputs": [],
      "source": [
        "experiment_run = experiment_df.run_name.iloc[0]\n",
        "\n",
        "with vertex_ai.start_run(experiment_run, resume=True) as run:\n",
        "    # get the latest logged custom job\n",
        "    logged_job = run.get_logged_custom_jobs()[-1]\n",
        "\n",
        "print(logged_job.job_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# delete flags\n",
        "delete_experiment = False\n",
        "delete_bucket = False\n",
        "\n",
        "# Delete experiment\n",
        "if delete_experiment or os.getenv(\"IS_TESTING\"):\n",
        "    experiment = vertex_ai.Experiment.get(experiment_name=EXPERIMENT_NAME)\n",
        "    experiment.delete(delete_backing_tensorboard_runs=True)\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_with_custom_training_autologging_local_script.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}