notebooks/official/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Vertex AI Pipelines: Evaluating batch prediction results from AutoML Tabular regression model\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/automl_tabular_regression_model_evaluation.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/automl_tabular_regression_model_evaluation.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/automl_tabular_regression_model_evaluation.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>                                                                                               \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to use Vertex AI regression model evaluation component to evaluate an AutoML Tabular regression model. Model evaluation helps you determine your model performance based on the evaluation metrics and improve the model if necessary. \n",
        "\n",
        "Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction). Learn more about [Regression for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to evaluate a Vertex AI model resource through a Vertex AI pipeline job using `google_cloud_pipeline_components`:\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Datasets (Tabular)\n",
        "- Vertex AI Training (AutoML Tabular Training)\n",
        "- Vertex AI Batch predictions\n",
        "- Vertex AI Pipelines\n",
        "- Vertex AI Model Registry\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create a Vertex AI Dataset\n",
        "- Configure a `AutoMLTabularTrainingJob`\n",
        "- Run the `AutoMLTabularTrainingJob` which returns a model\n",
        "- Import a pre-trained `AutoML model resource` into the pipeline\n",
        "- Run a `batch prediction` job in the pipeline\n",
        "- Evaluate the AutoML model using the `regression evaluation component`\n",
        "- Import the Regression Metrics to the AutoML model resource"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset being used in this notebook is a part of the PetFinder Dataset, available [here](https://www.kaggle.com/c/petfinder-adoption-prediction) on Kaggle. The current dataset is only a part of the original dataset considered for the problem of predicting age of the pet. It consists of the following fields:\n",
        "\n",
        "- `Type`: Type of animal (1 = Dog, 2 = Cat)\n",
        "- `Age`: Age of pet when listed, in months\n",
        "- `Breed1`: Primary breed of pet\n",
        "- `Gender`: Gender of pet\n",
        "- `Color1`: Color 1 of pet \n",
        "- `Color2`: Color 2 of pet\n",
        "- `MaturitySize`: Size at maturity (1 = Small, 2 = Medium, 3 = Large, 4 = Extra Large, 0 = Not Specified)\n",
        "- `FurLength`: Fur length (1 = Short, 2 = Medium, 3 = Long, 0 = Not Specified)\n",
        "- `Vaccinated`: Pet has been vaccinated (1 = Yes, 2 = No, 3 = Not Sure)\n",
        "- `Sterilized`: Pet has been spayed / neutered (1 = Yes, 2 = No, 3 = Not Sure)\n",
        "- `Health`: Health Condition (1 = Healthy, 2 = Minor Injury, 3 = Serious Injury, 0 = Not Specified)\n",
        "- `Fee`: Adoption fee (0 = Free)\n",
        "- `PhotoAmt`: Total uploaded photos for this pet\n",
        "- `Adopted`: Whether or not the pet was adopted (Yes/No).\n",
        "\n",
        "**Note**: This dataset is moved to a public Cloud Storage bucket and is accessed from there in this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs \n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
        "                                 google-cloud-pipeline-components==1.0.26 \\\n",
        "                                 matplotlib "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D-ZBOjErv5mM"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2dw8q9fdQEH5"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"\n",
        "DATA_REGION = \"US\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
        "\n",
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated.\n",
        "\n",
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ce6043da7b33"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0367eac06a10"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "21ad4dbb4a61"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c13224697bfb"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "autoset_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91c46850b49b"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "#### Service Account\n",
        "\n",
        "You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UwC1AdGeF6kx"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    else:  # IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6OqzKqhMF6kx"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XoEqT2Y4DJmf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pRUOFELefqf1"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import google.cloud.aiplatform as aiplatform\n",
        "import matplotlib.pyplot as plt\n",
        "from google.cloud import aiplatform_v1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ksAefQcCF6ky"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiVlyW5OUnjK"
      },
      "source": [
        "## Create Vertex AI Dataset\n",
        "\n",
        "Create a managed tabular dataset resource in Vertex AI using the dataset source."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bViYfWfpVAiF"
      },
      "outputs": [],
      "source": [
        "DATA_SOURCE = \"gs://cloud-samples-data/ai-platform-unified/datasets/tabular/petfinder-tabular-classification.csv\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "20S9En09X0PY"
      },
      "outputs": [],
      "source": [
        "# Create the Vertex AI Dataset resource\n",
        "dataset = aiplatform.TabularDataset.create(\n",
        "    display_name=\"petfinder-tabular-dataset\",\n",
        "    gcs_source=DATA_SOURCE,\n",
        ")\n",
        "\n",
        "print(\"Resource name:\", dataset.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A-QQkeUnq8Xt"
      },
      "source": [
        "## Train AutoML model\n",
        "\n",
        "Train a simple regression model using the created dataset using `Age` as the target column. \n",
        "\n",
        "**Set a display name and create the `AutoMLTabularTrainingJob` with appropriate data types specified for column transformations.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bxn6ATUXrET6"
      },
      "outputs": [],
      "source": [
        "TRAINING_JOB_DISPLAY_NAME = \"[your-train-job-display-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2e7664fe3af6"
      },
      "outputs": [],
      "source": [
        "# If no display name is specified, use the default one\n",
        "if (\n",
        "    TRAINING_JOB_DISPLAY_NAME == \"\"\n",
        "    or TRAINING_JOB_DISPLAY_NAME is None\n",
        "    or TRAINING_JOB_DISPLAY_NAME == \"[your-train-job-display-name]\"\n",
        "):\n",
        "    TRAINING_JOB_DISPLAY_NAME = \"train-pet-agefinder-automl\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6cb41277f4f3"
      },
      "source": [
        "### Define AutoML Tabular training job\n",
        "\n",
        "An AutoML training job is created with the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the `TrainingJob` resource.\n",
        "- `optimization_prediction_type`: The type of prediction the AutoML Model is to produce. Ex: regression, classification.\n",
        "- `column_transformations`: Transformations to apply to the input columns (i.e. columns other than the targetColumn). Each transformation may produce multiple result values from the column's value, and all are used for training. \n",
        "- `optimization_objective`: The optimization objective to minimize or maximize.\n",
        "    - `minimize-rmse`\n",
        "    - `minimize-mae`\n",
        "    - `minimize-rmsle`\n",
        "\n",
        "Learn about [AutoMLTabularTrainingJob](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.AutoMLTabularTrainingJob)    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3l691PEMZFdA"
      },
      "outputs": [],
      "source": [
        "train_job = aiplatform.AutoMLTabularTrainingJob(\n",
        "    display_name=TRAINING_JOB_DISPLAY_NAME,\n",
        "    optimization_prediction_type=\"regression\",\n",
        "    column_specs={\n",
        "        \"Type\": \"categorical\",\n",
        "        \"Breed1\": \"categorical\",\n",
        "        \"Gender\": \"categorical\",\n",
        "        \"Color1\": \"categorical\",\n",
        "        \"Color2\": \"categorical\",\n",
        "        \"MaturitySize\": \"categorical\",\n",
        "        \"FurLength\": \"categorical\",\n",
        "        \"Vaccinated\": \"categorical\",\n",
        "        \"Sterilized\": \"categorical\",\n",
        "        \"Health\": \"categorical\",\n",
        "        \"Fee\": \"numeric\",\n",
        "        \"PhotoAmt\": \"numeric\",\n",
        "        \"Adopted\": \"categorical\",\n",
        "    },\n",
        "    optimization_objective=\"minimize-rmse\",\n",
        ")\n",
        "\n",
        "print(train_job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "391c51c98647"
      },
      "source": [
        "#### Set the display name for the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "454f077b984e"
      },
      "outputs": [],
      "source": [
        "MODEL_DISPLAY_NAME = \"[your-model-display-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c4f338cdea7c"
      },
      "outputs": [],
      "source": [
        "# If no name is specified, use the default name\n",
        "if (\n",
        "    MODEL_DISPLAY_NAME == \"\"\n",
        "    or MODEL_DISPLAY_NAME is None\n",
        "    or MODEL_DISPLAY_NAME == \"[your-model-display-name]\"\n",
        "):\n",
        "    MODEL_DISPLAY_NAME = \"pet-agefinder-prediction-model\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de7e24205889"
      },
      "source": [
        "#### Run the training job\n",
        "\n",
        "Next, you start the training job by invoking the method `run`, with the following parameters:\n",
        "\n",
        "- `dataset`: The `Dataset` resource to train the model.\n",
        "- `target_column`: The name of the column, whose values the model is to predict.\n",
        "- `training_fraction_split`: The percentage of the dataset to use for training.\n",
        "- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
        "- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
        "- `model_display_name`: The human readable name for the trained model.\n",
        "- `budget_milli_node_hours`: The train budget of creating this Model, expressed in milli node hours i.e. 1,000 value in this field means 1 node hour. \n",
        "\n",
        "The training job takes roughly 3 hours to finish."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IIfvPCGYyFCT"
      },
      "outputs": [],
      "source": [
        "# Run the training job\n",
        "model = train_job.run(\n",
        "    dataset=dataset,\n",
        "    target_column=\"Age\",\n",
        "    training_fraction_split=0.8,\n",
        "    validation_fraction_split=0.1,\n",
        "    test_fraction_split=0.1,\n",
        "    model_display_name=MODEL_DISPLAY_NAME,\n",
        "    budget_milli_node_hours=1000,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYirKB_9yaa0"
      },
      "source": [
        "## List model evaluations from training\n",
        "\n",
        "After the training job is finished, get the model evaluations and print them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KkgCdQQAyZP1"
      },
      "outputs": [],
      "source": [
        "# Get evaluations\n",
        "model_evaluations = model.list_model_evaluations()\n",
        "\n",
        "model_evaluation = list(model_evaluations)[0]\n",
        "print(model_evaluation)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3f4d0c17150d"
      },
      "outputs": [],
      "source": [
        "# Print the evaluation metrics\n",
        "for evaluation in model_evaluations:\n",
        "    evaluation = evaluation.to_dict()\n",
        "    print(\"Model's evaluation metrics from Training:\\n\")\n",
        "    metrics = evaluation[\"metrics\"]\n",
        "    for metric in metrics.keys():\n",
        "        print(f\"metric: {metric}, value: {metrics[metric]}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2241f3739e03"
      },
      "source": [
        "## Run a pipeline for model evaluation\n",
        "\n",
        "Now, you run a Vertex AI BatchPrediction job and generate evaluations and feature-attributions on its results by creating a Vertex AI pipeline using `evaluate` function. Learn more about [evaluate function](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/models.py#L5127)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d17a0268020f"
      },
      "source": [
        "### Define parameters to run the evaluate function\n",
        "\n",
        "Specify the required parameters to run `evaluate` function. \n",
        "\n",
        "The following is the instruction of `evaluate` function paramters:\n",
        "\n",
        "- `prediction_type`: The problem type being addressed by this evaluation run. 'classification' and 'regression' are the currently supported problem types.\n",
        "- `target_field_name`: Name of the column to be used as the target for regression.\n",
        "- `gcs_source_uris`: List of the Cloud Storage bucket uris of input instances for batch prediction.\n",
        "- `generate_feature_attributions`: Optional. Whether the model evaluation job should generate feature attributions. Defaults to False if not specified.\n",
        "\n",
        "**The pipeline takes about 2 hours to complete.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9e41d48d1f48"
      },
      "outputs": [],
      "source": [
        "job = model.evaluate(\n",
        "    prediction_type=\"regression\",\n",
        "    target_field_name=\"Age\",\n",
        "    gcs_source_uris=[DATA_SOURCE],\n",
        "    generate_feature_attributions=True,\n",
        ")\n",
        "\n",
        "print(\"Waiting model evaluation is in process\")\n",
        "job.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2zocUvk2YVs"
      },
      "source": [
        "In the results from last step, click on the generated link to see your run in the Cloud Console.\n",
        "\n",
        "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",
        "<img src=\"images/automl_tabular_regression_evaluation_pipeline.PNG\" style=\"height:622px;width:726px\"></img>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcKaONSsGNC4"
      },
      "source": [
        "### Get the Model Evaluation Results\n",
        "\n",
        "After the evalution pipeline is finished, run the below cell to print the evaluation metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "80d4f5b11d24"
      },
      "outputs": [],
      "source": [
        "model_evaluation = job.get_model_evaluation()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mtHA8rhGGQv3"
      },
      "outputs": [],
      "source": [
        "# Iterate over the pipeline tasks\n",
        "for (\n",
        "    task\n",
        ") in model_evaluation._backing_pipeline_job._gca_resource.job_detail.task_details:\n",
        "    # Obtain the artifacts from the evaluation task\n",
        "    if (\n",
        "        (\"model-evaluation\" in task.task_name)\n",
        "        and (\"model-evaluation-import\" not in task.task_name)\n",
        "        and (\n",
        "            task.state == aiplatform_v1.types.PipelineTaskDetail.State.SUCCEEDED\n",
        "            or task.state == aiplatform_v1.types.PipelineTaskDetail.State.SKIPPED\n",
        "        )\n",
        "    ):\n",
        "        evaluation_metrics = task.outputs.get(\"evaluation_metrics\").artifacts[\n",
        "            0\n",
        "        ]  # ['artifacts']\n",
        "        evaluation_metrics_gcs_uri = evaluation_metrics.uri\n",
        "\n",
        "print(evaluation_metrics)\n",
        "print(evaluation_metrics_gcs_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e69f183f902b"
      },
      "source": [
        "### Visualize the metrics\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14MBD57k0Fng"
      },
      "source": [
        "After the evalution pipeline is finished, run the below cell to visualize the evaluation metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b7c5e5c35ee9"
      },
      "outputs": [],
      "source": [
        "metrics = []\n",
        "values = []\n",
        "for i in evaluation_metrics.metadata.items():\n",
        "    if (\n",
        "        i[0] == \"meanAbsolutePercentageError\"\n",
        "    ):  # we are not considering MAPE as it is infinite. MAPE is infinite if groud truth is 0 as in our case Age is 0 for some instances.\n",
        "        continue\n",
        "    metrics.append(i[0])\n",
        "    values.append(i[1])\n",
        "plt.figure(figsize=(10, 5))\n",
        "plt.bar(x=metrics, height=values)\n",
        "plt.title(\"Evaluation Metrics\")\n",
        "plt.ylabel(\"Value\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c26ad3958895"
      },
      "source": [
        "### Get the Feature Attributions\n",
        "\n",
        "Feature attributions indicate how much each feature in your model contributed to the predictions for each given instance.\n",
        "\n",
        "Learn more about [Feature Attributions](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview#feature_attributions)\n",
        "\n",
        "Run the below cell to get the feature attributions. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b09056628b26"
      },
      "outputs": [],
      "source": [
        "# Iterate over the pipeline tasks\n",
        "for (\n",
        "    task\n",
        ") in model_evaluation._backing_pipeline_job._gca_resource.job_detail.task_details:\n",
        "    # Obtain the artifacts from the feature-attribution task\n",
        "    if (task.task_name == \"feature-attribution\") and (\n",
        "        task.state == aiplatform_v1.types.PipelineTaskDetail.State.SUCCEEDED\n",
        "        or task.state == aiplatform_v1.types.PipelineTaskDetail.State.SKIPPED\n",
        "    ):\n",
        "        feat_attrs = task.outputs.get(\"feature_attributions\").artifacts[0]\n",
        "        feat_attrs_gcs_uri = feat_attrs.uri\n",
        "\n",
        "print(feat_attrs)\n",
        "print(feat_attrs_gcs_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4d9d6a82d826"
      },
      "source": [
        "From the obtained Cloud Storage uri for the feature attributions, get the attribution values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c26a2091f4fc"
      },
      "outputs": [],
      "source": [
        "# Load the results\n",
        "attributions = !gsutil cat $feat_attrs_gcs_uri\n",
        "\n",
        "# Print the results obtained\n",
        "attributions = json.loads(attributions[0])\n",
        "print(attributions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77151be8d776"
      },
      "source": [
        "### Visualize the Feature Attributions\n",
        "\n",
        "Visualize the obtained attributions for each feature using a bar-chart."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "069bf017e0de"
      },
      "outputs": [],
      "source": [
        "data = attributions[\"explanation\"][\"attributions\"][0][\"featureAttributions\"]\n",
        "features = []\n",
        "attr_values = []\n",
        "for key, value in data.items():\n",
        "    features.append(key)\n",
        "    attr_values.append(value)\n",
        "\n",
        "plt.figure(figsize=(5, 3))\n",
        "plt.bar(x=features, height=attr_values)\n",
        "plt.title(\"Feature Attributions\")\n",
        "plt.xticks(rotation=90)\n",
        "plt.ylabel(\"Attribution value\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial.\n",
        "\n",
        "Set `delete_bucket` to **True** to create the Cloud Storage bucket created in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Delete model resource\n",
        "model.delete()\n",
        "\n",
        "# Delete the dataset resource\n",
        "dataset.delete()\n",
        "\n",
        "# Delete the training job\n",
        "train_job.delete()\n",
        "\n",
        "# Delete the evaluation pipeline\n",
        "job.delete()\n",
        "\n",
        "# Delete Cloud Storage objects\n",
        "delete_bucket = False\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "automl_tabular_regression_model_evaluation.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}