notebooks/official/workbench/chicago_taxi_fare_prediction/chicago_taxi_fare_prediction.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c8c4e360024a"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6728c7e34d2"
      },
      "source": [
        "# Taxi fare prediction using the Chicago Taxi Trips dataset\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/chicago_taxi_fare_prediction/chicago_taxi_fare_prediction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/chicago_taxi_fare_prediction/chicago_taxi_fare_prediction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/workbench/chicago_taxi_fare_prediction/chicago_taxi_fare_prediction.ipynb\" target='_blank'>\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>                                                                                               \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8e07a1ab9882"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates analysis, feature selection, model building, and deployment with Explainable AI configured on Vertex AI, using a subset of the Chicago Taxi Trips dataset for taxi-fare prediction.\n",
        "\n",
        "*Note: This notebook is developed to run in a [Vertex AI Workbench managed notebooks](https://console.cloud.google.com/vertex-ai/workbench/list/managed) instance using the Python (Local) kernel. Some components of this notebook may not work in other notebook environments.*\n",
        "\n",
        "Learn more about [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction) and [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aa46a1d2695e"
      },
      "source": [
        "### Objective\n",
        "\n",
        "The goal of this notebook is to provide an overview on the latest Vertex AI features like **Explainable AI** and **BigQuery in Notebooks** by trying to solve a taxi fare prediction problem. \n",
        "\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- `Vertex AI Model` resource\n",
        "- `Vertex AI Endpoint` resource\n",
        "- `Vertex Explainable AI`\n",
        "-  Google Cloud Storage\n",
        "-  BigQuery\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Loading the dataset using \"BigQuery in Notebooks\".\n",
        "- Performing exploratory data analysis on the dataset.\n",
        "- Feature selection and preprocessing.\n",
        "- Building a linear regression model using scikit-learn.\n",
        "- Configuring the model for Vertex Explainable AI.\n",
        "- Deploying the model to Vertex AI.\n",
        "- Testing the deployed model.\n",
        "- Clean up."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9fff5c86f300"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The Chicago Taxi Trips dataset includes taxi trips from 2013 to the present, reported to the city of Chicago in its role as a regulatory agency. To protect privacy but allow for aggregate analyses, the taxi ID is consistent for any given taxi medallion number but does not show the number, census tracts are suppressed in some cases, and times are rounded to the nearest 15 minutes. Due to the data reporting process, not all trips are reported but the city believes that most are. This dataset is publicly available on BigQuery as a public dataset with the table ID `bigquery-public-data.chicago_taxi_trips.taxi_trips` and also as a public dataset on Kaggle at [Chicago Taxi Trips](https://www.kaggle.com/chicago/chicago-taxi-trips-bq).\n",
        "\n",
        "For more information about this dataset and how it was created, see the [Chicago Digital website](http://digital.cityofchicago.org/index.php/chicago-taxi-data-released).\n",
        "\n",
        "The original dataset considered for this tutorial is a large and noisy one and so data from a specific date range will be used. Based on various online resources, the data from around May 2018 gave some really good results compared to the other date ranges. While there are also some complicated models proposed for the same problem, like considering the weather data, holidays and seasons, the current notebook only explores a simple linear regression model. Our main objective is to demonstrate the model deployment with Vertex Explainable AI configured on Vertex AI.\n",
        "\n",
        "The chosen dataset consists of the following fields:\n",
        "\n",
        "- `unique_key` : Unique identifier for the trip.\n",
        "- `taxi_id` : A unique identifier for the taxi.\n",
        "- `trip_start_timestamp`: When the trip started, rounded to the nearest 15 minutes.\n",
        "- `trip_end_timestamp`: When the trip ended, rounded to the nearest 15 minutes.\n",
        "- `trip_seconds`: Time of the trip in seconds.\n",
        "- `trip_miles`: Distance of the trip in miles.\n",
        "- `pickup_census_tract`: The Census Tract where the trip began. For privacy, this Census Tract is not shown for some trips.\n",
        "- `dropoff_census_tract`: The Census Tract where the trip ended. For privacy, this Census Tract is not shown for some trips.\n",
        "- `pickup_community_area`: The Community Area where the trip began.\n",
        "- `dropoff_community_area`: The Community Area where the trip ended.\n",
        "- `fare`: The fare for the trip.\n",
        "- `tips`: The tip for the trip. Cash tips generally will not be recorded.\n",
        "- `tolls`: The tolls for the trip.\n",
        "- `extras`: Extra charges for the trip.\n",
        "- `trip_total`: Total cost of the trip, the total of the fare, tips, tolls, and extras.\n",
        "- `payment_type`: Type of payment for the trip.\n",
        "- `company`: The taxi company.\n",
        "- `pickup_latitude`: The latitude of the center of the pickup census tract or the community area if the census tract has been hidden for privacy.\n",
        "- `pickup_longitude`: The longitude of the center of the pickup census tract or the community area if the census tract has been hidden for privacy.\n",
        "- `pickup_location`: The location of the center of the pickup census tract or the community area if the census tract has been hidden for privacy.\n",
        "- `dropoff_latitude`: The latitude of the center of the dropoff census tract or the community area if the census tract has been hidden for privacy.\n",
        "- `dropoff_longitude`: The longitude of the center of the dropoff census tract or the community area if the census tract has been hidden for privacy.\n",
        "- `dropoff_location`: The location of the center of the dropoff census tract or the community area if the census tract has been hidden for privacy.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12e1391884e6"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses the following billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI\n",
        "- BigQuery\n",
        "- Cloud Storage\n",
        "\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing), [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23976b1be293"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1fd00fa70a2a"
      },
      "outputs": [],
      "source": [
        "! pip3 install --quiet --upgrade    google-cloud-bigquery \\\n",
        "                                    google-cloud-aiplatform \\\n",
        "                                    google-cloud-storage \\\n",
        "                                    seaborn==0.12.0 \\\n",
        "                                    scikit-learn \\\n",
        "                                    pandas \\\n",
        "                                    fsspec \\\n",
        "                                    db-dtypes \\\n",
        "                                    pyarrow   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Set the region\n",
        "\n",
        "**Optional**: Update the 'REGION' variable to specify the region that you want to use. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nsN5NJKSu-GU"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0c9906f72b18"
      },
      "source": [
        "### UUID\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8940c46120e6"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of length 8\n",
        "def generate_uuid():\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=8))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "To authenticate your Google Cloud account, follow the instructions for your Jupyter environment:\n",
        "\n",
        "**1. Vertex AI Workbench**\n",
        "<br>You are already authenticated.\n",
        "\n",
        "**2. Local JupyterLab instance**\n",
        "<br>Uncomment and run the following code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "254614fa0c46"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef21552ccea8"
      },
      "source": [
        "**3. Colab**\n",
        "<br>Uncomment and run the following code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "603adbbf0532"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c13224697bfb"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2e52fd6d4854"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "94d579ea1834"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pickle\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# load the required libraries\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "from google.cloud import aiplatform, storage\n",
        "from google.cloud.aiplatform_v1.types import SampledShapleyAttribution\n",
        "from google.cloud.aiplatform_v1.types.explanation import ExplanationParameters\n",
        "from google.cloud.bigquery import Client\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.metrics import mean_squared_error, r2_score\n",
        "from sklearn.model_selection import train_test_split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5166f42557ad"
      },
      "source": [
        "## Accessing the data through BigQuery Integration\n",
        "\n",
        "The **BigQuery Integration for Notebooks** feature of Vertex AI Workbench managed notebooks lets you use BigQuery and its features from the notebook itself eliminating the need to switch between tabs everytime. For every cell in the notebook, there is an option for the BigQuery integration at the top right, and selecting it enables you to compose an SQL query that can be executed in BigQuery. \n",
        "\n",
        "\n",
        "Among the available fields in the dataset, only the fields that seem common and relevant for analysis and modeling like `taxi_id`, `trip_start_timestamp`, `trip_seconds`, `trip_miles`, `payment_type` and `trip_total` are selected. Further, the field `trip_total` is treated as the target variable that would be predicted by the machine learning model. Apparently, this field is a summation of the `fare`,`tips`,`tolls` and `extras` fields and so because of their correlation with the target variable, they are being excluded for modeling. Due to the volume of the data, a subset of the dataset over the course of one week, 12-May-2018 to 18-May-2018 is being considered. Within this date range itself, the datapoints can be noisy and so a few conditions like the following are considered: \n",
        "\n",
        "- Time taken for the trip > 0.\n",
        "- Distance covered during the trip > 0.\n",
        "- Total trip charges > 0 and\n",
        "- Pickup and dropoff areas are valid (not empty).\n",
        "\n",
        "Note: The below cell is a Bigquery Integration cell and can only execute on Vertex AI Workbench's managed instances. If your notebook environment is different, you can skip it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "53e0070caf3d"
      },
      "source": [
        "#@bigquery\n",
        "\n",
        "select \n",
        "-- select the required fields\n",
        "taxi_id, trip_start_timestamp, \n",
        "trip_seconds, trip_miles, trip_total, \n",
        "payment_type\n",
        "\n",
        "from `bigquery-public-data.chicago_taxi_trips.taxi_trips` \n",
        "where \n",
        "-- specify the required criteria\n",
        "trip_start_timestamp >= '2018-05-12' and \n",
        "trip_end_timestamp <= '2018-05-18' and\n",
        "trip_seconds > 0 and\n",
        "trip_miles > 0 and\n",
        "trip_total > 3 and\n",
        "pickup_community_area is not NULL and \n",
        "dropoff_community_area is not NULL\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "781341730c28"
      },
      "source": [
        "The BigQuery integration also lets you load the queried data into a pandas dataframe using the `Query and load as DataFrame` button. Clicking the button adds a new cell below that provides a code snippet to load the data into a dataframe."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1a09f0c233e9"
      },
      "source": [
        "#### Select the required fields"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "beab4629fa0c"
      },
      "outputs": [],
      "source": [
        "# The following two lines are only necessary to run once.\n",
        "# Comment out otherwise for speed-up.\n",
        "\n",
        "client = Client(project=PROJECT_ID)\n",
        "\n",
        "query = \"\"\"select \n",
        "taxi_id, trip_start_timestamp, \n",
        "trip_seconds, trip_miles, trip_total, \n",
        "payment_type, pickup_community_area, \n",
        "dropoff_community_area \n",
        "\n",
        "from `bigquery-public-data.chicago_taxi_trips.taxi_trips` \n",
        "where \n",
        "trip_start_timestamp >= '2018-05-12' and \n",
        "trip_end_timestamp <= '2018-05-18' and\n",
        "trip_seconds > 60 and trip_seconds < 6*60*60 and\n",
        "trip_miles > 0 and\n",
        "trip_total > 3 and\n",
        "pickup_community_area is not NULL and \n",
        "dropoff_community_area is not NULL\"\"\"\n",
        "job = client.query(query)\n",
        "df = job.to_dataframe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96d61011e159"
      },
      "source": [
        "#### Check the fields in the data and their shape."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d99448551de8"
      },
      "outputs": [],
      "source": [
        "# check the dataframe's shape\n",
        "print(df.shape)\n",
        "# check the columns in the dataframe\n",
        "df.columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6a8b2bbc2ed"
      },
      "source": [
        "#### Check some sample data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "551f0d136f8d"
      },
      "outputs": [],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6f15942c4af7"
      },
      "source": [
        "#### Check the dtypes of fields in the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "328d4395da97"
      },
      "outputs": [],
      "source": [
        "df.dtypes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07e223a7261f"
      },
      "source": [
        "#### Check for null values in the dataframe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f4ff422f9ab6"
      },
      "outputs": [],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0feadc628e4"
      },
      "source": [
        "Depending on the percentage of null values in the data, one can choose to either drop them or impute them with mean/median (for numerical values) and mode (for categorical values). In the current data, there doesn't seem to be any null values.\n",
        "\n",
        "#### Check the numerical distributions of the fields (numerical). \n",
        "\n",
        "In case there are any fields with constant values, those fields can be dropped as they don't add any value to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91c411c8c3d9"
      },
      "outputs": [],
      "source": [
        "df.describe().T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9eb5c84d2efe"
      },
      "source": [
        "#### Identify the categorical and numerical fields in the data\n",
        "\n",
        "In the current dataset, `trip_total` is the target field. To access the fields by their type easily, identify the categorical and numerical fields in the data and save them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91deacb8f533"
      },
      "outputs": [],
      "source": [
        "target = \"trip_total\"\n",
        "categ_cols = [\"payment_type\", \"pickup_community_area\", \"dropoff_community_area\"]\n",
        "num_cols = [\"trip_seconds\", \"trip_miles\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9d6579050b8a"
      },
      "source": [
        "## Analyze numerical data\n",
        "\n",
        "To further anaylyze the data, there are various plots that can be used on numerical and categorical fields. In case of numerical data, you can use histograms and box plots. Bar charts are suited for categorical data to better understand the distribution of the data and the outliers in the data.\n",
        "\n",
        "#### Plot histograms and box plots on the numerical fields."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2066c6a1730c"
      },
      "outputs": [],
      "source": [
        "for i in num_cols + [target]:\n",
        "    _, ax = plt.subplots(1, 2, figsize=(12, 4))\n",
        "    df[i].plot(kind=\"hist\", bins=100, ax=ax[0])\n",
        "    ax[0].set_title(str(i) + \" -Histogram\")\n",
        "    df[i].plot(kind=\"box\", ax=ax[1])\n",
        "    ax[1].set_title(str(i) + \" -Boxplot\")\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3672976d67b"
      },
      "source": [
        "#### The field `trip_seconds` describes the time taken for the trip in seconds. For ease of our analysis, let us convert it into hours."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "245b8798305a"
      },
      "outputs": [],
      "source": [
        "df[\"trip_hours\"] = round(df[\"trip_seconds\"] / 3600, 2)\n",
        "df[\"trip_hours\"].plot(kind=\"box\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69980501e4cd"
      },
      "source": [
        "#### Similarly, another field `trip_speed` can be added by dividing `trip_miles` and `trip_hours` to understand the speed of the trip in miles/hour."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dc4d4e00e111"
      },
      "outputs": [],
      "source": [
        "df[\"trip_speed\"] = round(df[\"trip_miles\"] / df[\"trip_hours\"], 2)\n",
        "df[\"trip_speed\"].plot(kind=\"box\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58d57879aa8a"
      },
      "source": [
        "#### So far you've only looked at the univariate plots. To better understand the relationship between the variables, a pair-plot can be plotted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d8970a559dc3"
      },
      "outputs": [],
      "source": [
        "# generate a pairplot for 10K samples\n",
        "try:\n",
        "    sns.pairplot(\n",
        "        data=df[[\"trip_seconds\", \"trip_miles\", \"trip_total\", \"trip_speed\"]].sample(\n",
        "            10000\n",
        "        )\n",
        "    )\n",
        "    plt.show()\n",
        "except Exception as e:\n",
        "    print(e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b69e8094ba39"
      },
      "source": [
        "From the box plots and the histograms visualized so far, it is evident that there are some outliers causing skewness in the data which perhaps could be removed. Also, you can see some linear relationships between the independent variables considered in the pair-plot. For example, `trip_seconds` and `trip_miles` and the dependant variable `trip_total`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2c7cfd7204c"
      },
      "source": [
        "#### Restrict the data based on the following conditions to remove the outliers in the data to some extent :\n",
        "- Total charge being at least more than $3.\n",
        "- Total miles driven greater than 0 and less than 300 miles.\n",
        "- Total seconds driven at least 1 minute.\n",
        "- Total hours driven not more than 2 hours.\n",
        "- Speed of the trip not being more than 70 mph.\n",
        "\n",
        "These conditions are based on some general assumptions as clearly there were some recording errors like speed being greater than 500 mph and travel-time being more than 5 hours that led to outliers in the data. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1d011deaaec3"
      },
      "outputs": [],
      "source": [
        "# set constraints to remove outliers\n",
        "df = df[df[\"trip_total\"] > 3]\n",
        "\n",
        "df = df[(df[\"trip_miles\"] > 0) & (df[\"trip_miles\"] < 300)]\n",
        "\n",
        "df = df[df[\"trip_seconds\"] >= 60]\n",
        "\n",
        "df = df[df[\"trip_hours\"] <= 2]\n",
        "\n",
        "df = df[df[\"trip_speed\"] <= 70]\n",
        "df.reset_index(drop=True, inplace=True)\n",
        "df.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "341b581e2155"
      },
      "source": [
        "## Analyze categorical data\n",
        "\n",
        "#### Further, explore the categorical data by plotting the distribution of all the levels in each field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1c6179b8a438"
      },
      "outputs": [],
      "source": [
        "for i in categ_cols:\n",
        "    print(df[i].unique().shape)\n",
        "    df[i].value_counts(normalize=True).plot(kind=\"bar\", figsize=(10, 4))\n",
        "    plt.title(i)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a40a4b2d9d6a"
      },
      "source": [
        "From the above analysis, one can see that almost 99% of the transaction types are Cash and Credit Card. While there are also other type of transactions, their distribution is negligible. In such a case, the lower distribution levels can be dropped. On the other hand, the total number of pickup and dropoff community areas both seem to have the same levels which make sense. In this case also, one can choose to omit the lower distribution levels but you'd have to make sure that both the fields have the same levels afterward. In the current notebook, keep them as is and proceed with the modeling.\n",
        "\n",
        "The relationships between the target variable and the categorical fields can be represented through box plots. For each level, the corresponding distribution of the target variable can be identified."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "203598ed5098"
      },
      "outputs": [],
      "source": [
        "for i in categ_cols:\n",
        "    plt.figure(figsize=(10, 4))\n",
        "    sns.boxplot(x=i, y=target, data=df)\n",
        "    plt.xticks(rotation=45)\n",
        "    plt.title(i)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f49125a8a866"
      },
      "source": [
        "There seems to be one case where the `trip_total` is over 3000 and has the same pickup and dropoff community area: 28 is clearly an outlier compared to the rest of the points. This datapoint can be removed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "59cfd540ec3d"
      },
      "outputs": [],
      "source": [
        "df = df[df[\"trip_total\"] < 3000].reset_index(drop=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1315e5155505"
      },
      "source": [
        "#### Keep only the `Credit Card` and `Cash` payment types. Further, encode them by assigning 0 for `Credit Card` and 1 for `Cash` payment types."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e6a0dbe01b61"
      },
      "outputs": [],
      "source": [
        "# add payment_type\n",
        "df = df[df[\"payment_type\"].isin([\"Credit Card\", \"Cash\"])].reset_index(drop=True)\n",
        "# encode the payment types\n",
        "df[\"payment_type\"] = df[\"payment_type\"].apply(\n",
        "    lambda x: 0 if x == \"Credit Card\" else (1 if x == \"Cash\" else None)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58a1d9f0a122"
      },
      "source": [
        "#### There are also useful timestamp fields in the data. `trip_start_timestamp` represents the start timestamp of the taxi trip and fields like what day of week it was and what hour it was can be derived from it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7c5baac285ea"
      },
      "outputs": [],
      "source": [
        "df[\"trip_start_timestamp\"] = pd.to_datetime(df[\"trip_start_timestamp\"])\n",
        "df[\"dayofweek\"] = df[\"trip_start_timestamp\"].dt.dayofweek\n",
        "df[\"hour\"] = df[\"trip_start_timestamp\"].dt.hour"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "30ae02a15aa1"
      },
      "source": [
        "Since the current dataset is limited to only a week, if there isn't much variation in the newly derived fields with respect to the target variable, they can be dropped.\n",
        "\n",
        "#### Plot sum and average of the `trip_total` with respect to the `dayofweek`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d51087f4221e"
      },
      "outputs": [],
      "source": [
        "# plot sum and average of trip_total w.r.t the dayofweek\n",
        "_, ax = plt.subplots(1, 2, figsize=(10, 4))\n",
        "df[[\"dayofweek\", \"trip_total\"]].groupby(\"dayofweek\").trip_total.sum().plot(\n",
        "    kind=\"bar\", ax=ax[0]\n",
        ")\n",
        "ax[0].set_title(\"Sum of trip_total\")\n",
        "df[[\"dayofweek\", \"trip_total\"]].groupby(\"dayofweek\").trip_total.mean().plot(\n",
        "    kind=\"bar\", ax=ax[1]\n",
        ")\n",
        "ax[1].set_title(\"Avg. of trip_total\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea69561b95c3"
      },
      "source": [
        "#### Plot sum and average of the `trip_total` with respect to the `hour`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "541b65e2a39a"
      },
      "outputs": [],
      "source": [
        "_, ax = plt.subplots(1, 2, figsize=(10, 4))\n",
        "df[[\"hour\", \"trip_total\"]].groupby(\"hour\").trip_total.sum().plot(kind=\"bar\", ax=ax[0])\n",
        "ax[0].set_title(\"Sum of trip_total\")\n",
        "df[[\"hour\", \"trip_total\"]].groupby(\"hour\").trip_total.mean().plot(kind=\"bar\", ax=ax[1])\n",
        "ax[1].set_title(\"Avg. of trip_total\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "739e985af704"
      },
      "source": [
        "As these plots don't seem to have constant figures with respect to the target variable across their levels, they can be considered for training. In fact, to simplify things these derived features can be bucketed into fewer levels.\n",
        "\n",
        "The `dayofweek` field can be bucketed into a binary field considering whether or not it was a weekend. If it is a weekday, the record can be assigned 1, else 0. Similarly, the `hour` field can also be bucketed and encoded. The normal working hours in Chicago can be assumed to be between *8AM*-*10PM* and if the value falls in between the working hours, it can be encoded as 1, else 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad00e88e98bb"
      },
      "outputs": [],
      "source": [
        "# bucket and encode the dayofweek and hour\n",
        "df[\"dayofweek\"] = df[\"dayofweek\"].apply(lambda x: 0 if x in [5, 6] else 1)\n",
        "df[\"hour\"] = df[\"hour\"].apply(lambda x: 0 if x in [23, 0, 1, 2, 3, 4, 5, 6, 7] else 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "884deb0356ab"
      },
      "source": [
        "#### Check the data distribution before training the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "597608af71e8"
      },
      "outputs": [],
      "source": [
        "df.describe().T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fe87612faa94"
      },
      "source": [
        "## Divide the data into train and test sets\n",
        "\n",
        "#### Split the preprocessed dataset into train and test sets so that the linear regression model can be validated on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0e2cee9d6322"
      },
      "outputs": [],
      "source": [
        "cols = [\n",
        "    \"trip_seconds\",\n",
        "    \"trip_miles\",\n",
        "    \"payment_type\",\n",
        "    \"pickup_community_area\",\n",
        "    \"dropoff_community_area\",\n",
        "    \"dayofweek\",\n",
        "    \"hour\",\n",
        "    \"trip_speed\",\n",
        "]\n",
        "x = df[cols].copy()\n",
        "y = df[target].copy()\n",
        "\n",
        "# split the data into 75-25% ratio\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    x, y, train_size=0.75, test_size=0.25, random_state=13\n",
        ")\n",
        "X_train.shape, X_test.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5b7e470de1da"
      },
      "source": [
        "## Fit a simple linear regression model\n",
        "\n",
        "#### Fit a linear regression model using scikit-learn's LinearRegression method on the train data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5195bdddf71d"
      },
      "outputs": [],
      "source": [
        "# Building the regression model\n",
        "reg = LinearRegression()\n",
        "reg.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7589e2e3e31d"
      },
      "source": [
        "#### Print the `R2 score` and `RMSE` values for the model on train and test sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4aa6843df9df"
      },
      "outputs": [],
      "source": [
        "# print test R2 score\n",
        "y_train_pred = reg.predict(X_train)\n",
        "train_score = r2_score(y_train, y_train_pred)\n",
        "train_rmse = mean_squared_error(y_train, y_train_pred, squared=False)\n",
        "y_test_pred = reg.predict(X_test)\n",
        "test_score = r2_score(y_test, y_test_pred)\n",
        "test_rmse = mean_squared_error(y_test, y_test_pred, squared=False)\n",
        "print(\"Train R2-score:\", train_score, \"Train RMSE:\", train_rmse)\n",
        "print(\"Test R2-score:\", test_score, \"Test RMSE:\", test_rmse)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ef6b44f0f93"
      },
      "source": [
        "A low RMSE error and a train and test R2 score of 0.93 suggests that the model is fitted well. Further, the coefficients learned by the model for each of its independent variables can also be checked by checking the `coef_` attribute of the sklearn model. \n",
        "\n",
        "#### Check the coefficients learned by the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "facea9070f80"
      },
      "outputs": [],
      "source": [
        "coef_df = pd.DataFrame({\"col\": cols, \"coeff\": reg.coef_})\n",
        "coef_df.set_index(\"col\").plot(kind=\"bar\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcaed0b52e60"
      },
      "source": [
        "## Save the model and upload to a Cloud Storage bucket\n",
        "\n",
        "#### To deploy the model on Vertex AI, the model artifacts need to be stored in a Cloud Storage bucket first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "264a67fd8fc7"
      },
      "outputs": [],
      "source": [
        "FILE_NAME = \"model.pkl\"\n",
        "with open(FILE_NAME, \"wb\") as file:\n",
        "    pickle.dump(reg, file)\n",
        "\n",
        "# Upload the saved model file to Cloud Storage\n",
        "BLOB_PATH = \"taxicab_fare_prediction/\"\n",
        "\n",
        "BLOB_NAME = BLOB_PATH + FILE_NAME\n",
        "\n",
        "bucket = storage.Client().bucket(BUCKET_URI[5:])\n",
        "blob = bucket.blob(BLOB_NAME)\n",
        "blob.upload_from_filename(FILE_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9f8ecfa6a19b"
      },
      "source": [
        "## Deploy the model on Vertex AI with support for Vertex Explainable AI\n",
        "\n",
        "Configure Vertex Explainable AI before deploying the model. Learn more about [Configuring Vertex Explainable AI in Vertex AI models](https://cloud.google.com/vertex-ai/docs/explainable-ai/configuring-explanations#scikit-learn-and-xgboost-pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a130721a5375"
      },
      "outputs": [],
      "source": [
        "MODEL_DISPLAY_NAME = \"[your-model-display-name]\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4dc63c440d4b"
      },
      "outputs": [],
      "source": [
        "# If the model display name is not set, choose the default one\n",
        "if MODEL_DISPLAY_NAME == \"[your-model-display-name]\":\n",
        "    MODEL_DISPLAY_NAME = \"taxi_fare_prediction_model\"\n",
        "\n",
        "\n",
        "ARTIFACT_GCS_PATH = f\"{BUCKET_URI}/{BLOB_PATH}\"\n",
        "\n",
        "# Feature-name(Inp_feature) and Output-name(Model_output) can be arbitrary\n",
        "exp_metadata = {\"inputs\": {\"Input_feature\": {}}, \"outputs\": {\"Predicted_taxi_fare\": {}}}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49ac289687d8"
      },
      "source": [
        "#### Create a model resource from the uploaded model with explanation metadata configured."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6ab9d32ab6a2"
      },
      "outputs": [],
      "source": [
        "# Create a Vertex AI model resource with support for Vertex Explainable AI\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=REGION)\n",
        "\n",
        "model = aiplatform.Model.upload(\n",
        "    display_name=MODEL_DISPLAY_NAME,\n",
        "    artifact_uri=ARTIFACT_GCS_PATH,\n",
        "    serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest\",\n",
        "    explanation_metadata=exp_metadata,\n",
        "    explanation_parameters=ExplanationParameters(\n",
        "        sampled_shapley_attribution=SampledShapleyAttribution(path_count=25)\n",
        "    ),\n",
        ")\n",
        "\n",
        "model.wait()\n",
        "\n",
        "print(model.display_name)\n",
        "print(model.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ed1bd9f0957"
      },
      "source": [
        "### Create an Endpoint resource for the model\n",
        "\n",
        "#### Set a display name for the endpoint and create the endpoint resource."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f0e5cea786b4"
      },
      "outputs": [],
      "source": [
        "ENDPOINT_DISPLAY_NAME = \"[your-endpoint-display-name]\"  # @param {type: \"string\"}\n",
        "\n",
        "# If the display name is not set, choose the default one\n",
        "if ENDPOINT_DISPLAY_NAME == \"[your-endpoint-display-name]\":\n",
        "    ENDPOINT_DISPLAY_NAME = \"taxi_fare_prediction_endpoint\"\n",
        "\n",
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=ENDPOINT_DISPLAY_NAME, project=PROJECT_ID, location=REGION\n",
        ")\n",
        "\n",
        "print(endpoint.display_name)\n",
        "print(endpoint.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9eaab1c54d66"
      },
      "source": [
        "### Deploy the model to the created endpoint with the required machine type\n",
        "\n",
        "#### Set a name for the deployment and deploy the model to the created endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ea0f52526343"
      },
      "outputs": [],
      "source": [
        "DEPLOYED_MODEL_NAME = \"[your-deployed-model-name]\"  # @param {type: \"string\"}\n",
        "\n",
        "# If the deployment name is not set, choose the default one\n",
        "if DEPLOYED_MODEL_NAME == \"[your-deployed-model-name]\":\n",
        "    DEPLOYED_MODEL_NAME = \"taxi_fare_prediction_deployment\"\n",
        "\n",
        "# Set the machine type to n1-standard2\n",
        "MACHINE_TYPE = \"n1-standard-2\"\n",
        "\n",
        "# Deploy the model to the endpoint\n",
        "model.deploy(\n",
        "    endpoint=endpoint,\n",
        "    deployed_model_display_name=DEPLOYED_MODEL_NAME,\n",
        "    machine_type=MACHINE_TYPE,\n",
        ")\n",
        "\n",
        "model.wait()\n",
        "\n",
        "print(model.display_name)\n",
        "print(model.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "686cfdcbaef8"
      },
      "source": [
        "#### To ensure the model is deployed, the ID of the deployed model can be checked using the `endpoint.list_models()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bab07adf5339"
      },
      "outputs": [],
      "source": [
        "endpoint.list_models()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b751978ff665"
      },
      "source": [
        "## Get explanations from the deployed model\n",
        "\n",
        "#### For testing the deployed online model, select two instances from the test data as payload."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2424e7980c08"
      },
      "outputs": [],
      "source": [
        "# format the top 2 test instances as the request's payload\n",
        "test_json = {\"instances\": [X_test.iloc[0].tolist(), X_test.iloc[1].tolist()]}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "01532047a99e"
      },
      "source": [
        "Call the endpoint with the payload request and parse the response for explanations. The explanations consists of attributions on the independent variables used for training the model which are based on the configured attribution method. In this case, we've used the `Sampled Shapely` method which assigns credit for the outcome to each feature, and considers different permutations of the features. This method provides a sampling approximation of exact Shapely values. Further information on the attribution methods for explanations can be found at [Overview of Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6278ba230865"
      },
      "outputs": [],
      "source": [
        "features = X_train.columns.to_list()\n",
        "\n",
        "\n",
        "def plot_attributions(attrs):\n",
        "    \"\"\"\n",
        "    Function to plot the features and their attributions for an instance\n",
        "    \"\"\"\n",
        "    rows = {\"feature_name\": [], \"attribution\": []}\n",
        "    for i, val in enumerate(features):\n",
        "        rows[\"feature_name\"].append(val)\n",
        "        rows[\"attribution\"].append(attrs[\"Input_feature\"][i])\n",
        "    attr_df = pd.DataFrame(rows).set_index(\"feature_name\")\n",
        "    attr_df.plot(kind=\"bar\")\n",
        "    plt.show()\n",
        "    return\n",
        "\n",
        "\n",
        "def explain_tabular_sample(\n",
        "    project: str, location: str, endpoint_id: str, instances: list\n",
        "):\n",
        "    \"\"\"\n",
        "    Function to make an explanation request for the specified payload and generate feature attribution plots\n",
        "    \"\"\"\n",
        "    aiplatform.init(project=project, location=location)\n",
        "\n",
        "    # endpoint = aiplatform.Endpoint(endpoint_id)\n",
        "\n",
        "    response = endpoint.explain(instances=instances)\n",
        "    print(\"#\" * 10 + \"Explanations\" + \"#\" * 10)\n",
        "    for explanation in response.explanations:\n",
        "        print(\" explanation\")\n",
        "        # Feature attributions.\n",
        "        attributions = explanation.attributions\n",
        "\n",
        "        for attribution in attributions:\n",
        "            print(\"  attribution\")\n",
        "            print(\"   baseline_output_value:\", attribution.baseline_output_value)\n",
        "            print(\"   instance_output_value:\", attribution.instance_output_value)\n",
        "            print(\"   output_display_name:\", attribution.output_display_name)\n",
        "            print(\"   approximation_error:\", attribution.approximation_error)\n",
        "            print(\"   output_name:\", attribution.output_name)\n",
        "            output_index = attribution.output_index\n",
        "            for output_index in output_index:\n",
        "                print(\"   output_index:\", output_index)\n",
        "\n",
        "            plot_attributions(attribution.feature_attributions)\n",
        "\n",
        "    print(\"#\" * 10 + \"Predictions\" + \"#\" * 10)\n",
        "    for prediction in response.predictions:\n",
        "        print(prediction)\n",
        "\n",
        "    return response\n",
        "\n",
        "\n",
        "test_json = [X_test.iloc[0].tolist(), X_test.iloc[1].tolist()]\n",
        "prediction = explain_tabular_sample(PROJECT_ID, REGION, endpoint, test_json)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87cf259efb64"
      },
      "source": [
        "## Next steps\n",
        "\n",
        "Since the Chicago Taxi Trips dataset is continuously updating, one can preform the same kind of analysis and model training every time a new set of data is available. The date range can also be increased from a week to a month or more depending on the quality of the data. Most of the steps followed in this notebook would still be valid and can be applied over the new data unless the data is too noisy. In fact, the notebook itself can be scheduled to run at the specified times to retrain the model using the scheduling option of [Vertex AI Workbench's executor](https://console.cloud.google.com/vertex-ai/workbench/list/executions). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eae8d94e3641"
      },
      "source": [
        "## Clean up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "- Model\n",
        "- Endpoint\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "24d89ecfdd0f"
      },
      "outputs": [],
      "source": [
        "# Undeploy the model\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# Delete the endpoint resource\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete the model resource.\n",
        "model.delete()\n",
        "\n",
        "# Set this to true only if you'd like to delete your bucket\n",
        "delete_bucket = False\n",
        "\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "chicago_taxi_fare_prediction.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}