notebooks/official/explainable_ai/sdk_automl_tabular_binary_classification_batch_explain.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title"
      },
      "source": [
        "# Vertex AI SDK: Batch explanation for AutoML tabular binary classification model\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_automl_tabular_binary_classification_batch_explain.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fexplainable_ai%2Fsdk_automl_tabular_binary_classification_batch_explain.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/explainable_ai/sdk_automl_tabular_binary_classification_batch_explain.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_automl_tabular_binary_classification_batch_explain.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:automl,xai"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use the Vertex AI SDK to create a tabular binary classification model and do batch prediction with explanations using Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users).\n",
        "\n",
        "Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview). Learn more about [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:automl,training,batch_prediction,xai"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn to use `AutoML` to create a tabular binary classification model from a Python script, and then learn to use `Vertex AI Batch Prediction` to make predictions with explanations. You can alternatively create and deploy models using the `gcloud` command-line tool or online using the Cloud Console.\n",
        "\n",
        "This tutorial uses the following Vertex AI services:\n",
        "\n",
        "- Vertex AI AutoML\n",
        "- Vertex AI Batch Prediction\n",
        "- Vertex Explainable AI\n",
        "- Vertex AI Model resource\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create a Vertex AI managed dataset resource.\n",
        "- Train an AutoML tabular binary classification model.\n",
        "- View the model evaluation metrics for the trained model.\n",
        "- Make a batch prediction request with explainability.\n",
        "\n",
        "There is one key difference between using batch prediction and using online prediction:\n",
        "\n",
        "* **Prediction Service**: Does an on-demand prediction for the entire set of instances (i.e., one or more data items) and returns the results in real-time.\n",
        "\n",
        "* **Batch Prediction Service**: Does a queued (batch) prediction for the entire set of instances in the background and stores the results in a Cloud Storage bucket when ready."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a4881cf39a4"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses a bank marketing dataset. This dataset doesn't require any feature engineering. The version of the dataset used in this tutorial is stored in a public Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0316df526f8"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform\n",
        "! pip3 install --upgrade --quiet google-cloud-storage\n",
        "! pip3 install --upgrade --quiet tensorflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D-ZBOjErv5mM"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee775571c2b5"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92e68cfc3a90"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "46604f70e831"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c708f022953d"
      },
      "source": [
        "### Set Google Cloud project information \n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Initialize the Vertex AI SDK for Python using the location and staging bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tutorial_start:automl"
      },
      "source": [
        "## Take a quick peek at the dataset\n",
        "\n",
        "You use a version of the Bank Marketing dataset that's stored in a public Cloud Storage bucket, using a CSV index file."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_file:u_dataset,csv"
      },
      "source": [
        "Set the variable `IMPORT_FILE` to the Cloud Storage location of the CSV index file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_file:bank,csv,lbn"
      },
      "outputs": [],
      "source": [
        "IMPORT_FILE = \"gs://cloud-ml-tables-data/bank-marketing.csv\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4a5007bb4010"
      },
      "source": [
        "Count the number of examples by counting the number of rows in the CSV index file  (`wc -l`) and then peek at the first few rows.\n",
        "\n",
        "For training using AutoML, you need to know the title of the label column, which is later saved as `label_column`. In this dataset, it's saved as the last column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "quick_peek:tabular"
      },
      "outputs": [],
      "source": [
        "count = ! gsutil cat $IMPORT_FILE | wc -l\n",
        "print(\"Number of Examples\", int(count[0]))\n",
        "\n",
        "print(\"First 10 rows\")\n",
        "! gsutil cat $IMPORT_FILE | head\n",
        "\n",
        "heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
        "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
        "print(\"Label Column Name\", label_column)\n",
        "if label_column is None:\n",
        "    raise Exception(\"label column missing\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_dataset:tabular,lbn"
      },
      "source": [
        "### Create the Dataset\n",
        "\n",
        "Next, create the Vertex AI managed dataset resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the resource.\n",
        "- `gcs_source`: A list of one or more dataset index files to import the data items into the resource.\n",
        "- `bq_source`: Alternatively, import data items from a BigQuery table into the resource.\n",
        "\n",
        "This operation may take several minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_dataset:tabular,lbn"
      },
      "outputs": [],
      "source": [
        "dataset = aiplatform.TabularDataset.create(\n",
        "    display_name=\"Bank Marketing\", gcs_source=[IMPORT_FILE]\n",
        ")\n",
        "\n",
        "print(dataset.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_pipeline:tabular,lbn"
      },
      "source": [
        "### Create and run training pipeline\n",
        "\n",
        "To train the AutoML model on your dataset, you perform the following steps: \n",
        "1) Create a training pipeline.\n",
        "2) Run the pipeline.\n",
        "\n",
        "#### Create training pipeline\n",
        "\n",
        "An AutoML training pipeline is created using the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the `TrainingJob` resource.\n",
        "- `optimization_prediction_type`: The type task to train the model for.\n",
        "  - `classification`: A tabuar classification model.\n",
        "  - `regression`: A tabular regression model.\n",
        "- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
        "- `optimization_objective`: The optimization objective to minimize or maximize.\n",
        "  - For binary classification:\n",
        "    - `minimize-log-loss`\n",
        "    - `maximize-au-roc`\n",
        "    - `maximize-au-prc`\n",
        "    - `maximize-precision-at-recall`\n",
        "    - `maximize-recall-at-precision`\n",
        "  - For multi-class classification:\n",
        "    - `minimize-log-loss`\n",
        "  - For regression:\n",
        "    - `minimize-rmse`\n",
        "    - `minimize-mae`\n",
        "    - `minimize-rmsle`\n",
        "\n",
        "The instantiated object is a DAG (directed acyclic graph) for the training pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_pipeline:tabular,lbn"
      },
      "outputs": [],
      "source": [
        "dag = aiplatform.AutoMLTabularTrainingJob(\n",
        "    display_name=\"bank\",\n",
        "    optimization_prediction_type=\"classification\",\n",
        "    optimization_objective=\"minimize-log-loss\",\n",
        ")\n",
        "\n",
        "print(dag)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_automl_pipeline:tabular"
      },
      "source": [
        "#### Run the training pipeline\n",
        "\n",
        "Next, you run the DAG to start the training job by invoking the `run` method with the following parameters:\n",
        "\n",
        "- `dataset`: The managed dataset resource to train the model.\n",
        "- `model_display_name`: The human readable name for the trained model.\n",
        "- `training_fraction_split`: The percentage of the dataset to use for training.\n",
        "- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
        "- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
        "- `target_column`: The name of the column to train as the label.\n",
        "- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = 1 hour). The training cost of the model doesn't exceed this budget.\n",
        "- `disable_early_stopping`: If set to `True`, training maybe completed before using the entire budget if the service believes it can't further improve on the model objective measurements.\n",
        "\n",
        "The `run` method when completed returns the Vertex AI model resource.\n",
        "\n",
        "The execution of the training pipeline takes upto 20 minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_automl_pipeline:tabular"
      },
      "outputs": [],
      "source": [
        "model = dag.run(\n",
        "    dataset=dataset,\n",
        "    model_display_name=\"bank\",\n",
        "    training_fraction_split=0.6,\n",
        "    validation_fraction_split=0.2,\n",
        "    test_fraction_split=0.2,\n",
        "    budget_milli_node_hours=1000,\n",
        "    disable_early_stopping=False,\n",
        "    target_column=label_column,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evaluate_the_model:mbsdk"
      },
      "source": [
        "## Review model evaluation scores\n",
        "\n",
        "After your model training has finished, you can review the evaluation scores using the `list_model_evaluations()` method. This method returns an iterator for each evaluation slice."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evaluate_the_model:mbsdk"
      },
      "outputs": [],
      "source": [
        "model_evaluations = model.list_model_evaluations()\n",
        "\n",
        "for model_evaluation in model_evaluations:\n",
        "    print(model_evaluation.to_dict())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_prediction"
      },
      "source": [
        "## Send a batch prediction request\n",
        "\n",
        "Send a batch prediction to your registered model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_batch_file:automl,tabular,alt"
      },
      "source": [
        "### Prepare test data\n",
        "\n",
        "You use synthetic data as a test data items. Don't be concerned that we are using synthetic data.\n",
        "\n",
        "Make a batch input file, which is later stored in your Cloud Storage bucket. Unlike image, video and text, the batch input file for tabular model only supports CSV format. To prepare the CSV file, you make sure that:\n",
        "\n",
        "- The first line is the heading with the feature (fields) heading names.\n",
        "- Each remaining line is a separate prediction request with the corresponding feature values.\n",
        "\n",
        "For example:\n",
        "\n",
        "    \"feature_1\", \"feature_2\". ...\n",
        "    value_1, value_2, ..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_batch_file:automl,tabular,alt"
      },
      "outputs": [],
      "source": [
        "! gsutil cat $IMPORT_FILE | head -n 1 > tmp.csv\n",
        "! gsutil cat $IMPORT_FILE | tail -n 10 >> tmp.csv\n",
        "\n",
        "! cut -d, -f1-16 tmp.csv > batch.csv\n",
        "\n",
        "gcs_input_uri = BUCKET_URI + \"/test.csv\"\n",
        "\n",
        "! gsutil cp batch.csv $gcs_input_uri"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_explain_request:mbsdk,csv"
      },
      "source": [
        "### Make the batch explanation request\n",
        "\n",
        "As your model resource is created, you can make a batch prediction by invoking the `batch_predict()` method with the following parameters:\n",
        "\n",
        "- `job_display_name`: The human readable name for the batch prediction job.\n",
        "- `gcs_source`: A list of one or more batch request input files.\n",
        "- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
        "- `instances_format`: The format for the input instances, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
        "- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
        "- `generate_explanations`: Set to `True` to generate explanations.\n",
        "- `sync`: If set to True, the call waits until the batch job is completed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "batch_explain_request:mbsdk,csv"
      },
      "outputs": [],
      "source": [
        "batch_predict_job = model.batch_predict(\n",
        "    job_display_name=\"bank\",\n",
        "    gcs_source=gcs_input_uri,\n",
        "    gcs_destination_prefix=BUCKET_URI,\n",
        "    instances_format=\"csv\",\n",
        "    predictions_format=\"jsonl\",\n",
        "    generate_explanation=True,\n",
        "    sync=False,\n",
        ")\n",
        "\n",
        "print(batch_predict_job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "source": [
        "### Wait for completion of batch prediction job\n",
        "\n",
        "Next, wait for the batch job to complete. \n",
        "\n",
        "Alternatively, you can set `sync` to `True` in the `batch_predict()` method earlier to perform the wait operation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "outputs": [],
      "source": [
        "batch_predict_job.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_batch_explanation:mbsdk,lbn"
      },
      "source": [
        "### Get the explanations\n",
        "\n",
        "Next, get the explanation results from the completed batch prediction job.\n",
        "\n",
        "The results are written to the output Cloud Storage location you specified in the batch prediction request. \n",
        "\n",
        "Call the `iter_outputs()` method to get a list of each Cloud Storage file generated in the results. Each file contains one or more explanation responses in the specified prediction format i.e., JSONL format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_batch_explanation:mbsdk,lbn"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "bp_iter_outputs = batch_predict_job.iter_outputs()\n",
        "\n",
        "explanation_results = list()\n",
        "for blob in bp_iter_outputs:\n",
        "    if blob.name.split(\"/\")[-1].startswith(\"explanation\"):\n",
        "        explanation_results.append(blob.name)\n",
        "\n",
        "tags = list()\n",
        "for explanation_result in explanation_results:\n",
        "    gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{explanation_result}\"\n",
        "    with tf.io.gfile.GFile(name=gfile_name, mode=\"r\") as gfile:\n",
        "        for line in gfile.readlines():\n",
        "            print(line)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "# Delete the training job\n",
        "dag.delete()\n",
        "# Delete the tabular dataset\n",
        "dataset.delete()\n",
        "# Delete the model resource\n",
        "model.delete()\n",
        "# Delete the batch prediction job\n",
        "batch_predict_job.delete()\n",
        "\n",
        "# Set this to true only if you'd like to delete your bucket\n",
        "delete_bucket = False\n",
        "\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI\n",
        "\n",
        "# Delete the locally generated files\n",
        "! rm batch.csv tmp.csv"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "sdk_automl_tabular_binary_classification_batch_explain.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}