notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsv4jGuU89rX"
      },
      "source": [
        "# Train a TabNet model using Vertex AI Remote Training with Vertex AI SDK 2.0\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
        "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "    <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
        "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
        "    </a>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:automl"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI SDK 2.0 for remote TabNet model training of a local TabNet model training job."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cTelQGJjifdu"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn to use `Vertex AI SDK 2.0` to remotely train a TabNet model as a local (on-prem) training job.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- `Vertex AI Training`\n",
        "- `Vertex AI Remote Training`\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Download and split the dataset\n",
        "- Ingest the data in a Dataframe and perform transformations.\n",
        "- Train a tabular classification model.\n",
        "- Train a tabular regression model.\n",
        "\n",
        "\n",
        "**TabNet remote training**\n",
        "```\n",
        "from google.cloud.aiplatform.private_preview.vertex_ai.tabular_models import TabNetTrainer\n",
        "\n",
        "vertex_ai.init(remote=True, project=\"my-project\", location=\"my-location\", staging_bucket=\"gs://my-bucket\")\n",
        "\n",
        "# Instantiate TabNetTrainer\n",
        "tabnet_trainer = TabNetTrainer(...)\n",
        "\n",
        "# Optional set training config\n",
        "tabnet_trainer.fit.vertex.training_config.staging_bucket = \"gs://my-bucket\"\n",
        "\n",
        "# This `fit` call will be executed remotely\n",
        "tabnet_trainer.fit(...)\n",
        "\n",
        "# This `predict` call will be executed locally\n",
        "tabnet_trainer.predict(...)\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aef4f59195ad"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This tutorial uses the <a href=\"https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\">IRIS dataset</a>, which predicts the iris species."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F9dJ5Of-dORl"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform[preview] \\\n",
        "                                 scikit-learn \\\n",
        "                                 pyarrow \\\n",
        "                                 fastparquet \\\n",
        "                                 tensorflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D-ZBOjErv5mM"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2dw8q9fdQEH5"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
        "\n",
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated.\n",
        "\n",
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ce6043da7b33"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0367eac06a10"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "21ad4dbb4a61"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c13224697bfb"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "autoset_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91c46850b49b"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LF2bwT6q-of1"
      },
      "outputs": [],
      "source": [
        "import uuid\n",
        "\n",
        "import pandas as pd\n",
        "import vertexai.preview\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split\n",
        "from vertexai.preview.tabular_models import TabNetTrainer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-p0YAOY64cA5"
      },
      "outputs": [],
      "source": [
        "vertexai.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=REGION,\n",
        "    staging_bucket=BUCKET_URI,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "innAVhRdxe8c"
      },
      "source": [
        "### Prepare the dataset\n",
        "\n",
        "Now load the Iris dataset and split the data into train, evaluation and test sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HPT2EX7ii9jO"
      },
      "outputs": [],
      "source": [
        "dataset = load_iris()\n",
        "\n",
        "X, X_validation, y, y_validation = train_test_split(\n",
        "    dataset.data, dataset.target, test_size=0.60, random_state=42\n",
        ")\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.20, random_state=42\n",
        ")\n",
        "\n",
        "print(\"Data size: \", len(dataset.target))\n",
        "print(\"X_train size: \", len(X_train))\n",
        "print(\"X_validation size: \", len(X_validation))\n",
        "print(\"X_test size: \", len(X_test))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XzQfxEwuY22S"
      },
      "source": [
        "## Feature transformation\n",
        "\n",
        "Next, you do ingest the training and evaluation data in pandas Dataframe and perform feature transformations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sAHGB2nYjcDg"
      },
      "outputs": [],
      "source": [
        "columns = [f\"arg_{i}\" for i in range(X_train.shape[1])]\n",
        "\n",
        "training_data = pd.DataFrame(X_train, columns=columns)\n",
        "training_data[\"target\"] = y_train\n",
        "training_data[\"target\"] = training_data[\"target\"].astype(\"category\")\n",
        "\n",
        "validation_data = pd.DataFrame(X_validation, columns=columns)\n",
        "validation_data[\"target\"] = y_validation\n",
        "validation_data[\"target\"] = validation_data[\"target\"].astype(\"category\")\n",
        "\n",
        "eval_data = pd.DataFrame(X_test, columns=columns)\n",
        "\n",
        "training_data.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0kyuTJLli5eB"
      },
      "source": [
        "## Tabular classification model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LE4qd1T3kfpR"
      },
      "source": [
        "### Initialize TabNetTrainer\n",
        "\n",
        "Next, initialize the TabNetTrainer for training a tabular classification model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eGXmfieOklSs"
      },
      "outputs": [],
      "source": [
        "tabnet_trainer = TabNetTrainer(\n",
        "    model_type=\"classification\",\n",
        "    target_column=\"target\",\n",
        "    learning_rate=0.01,\n",
        "    max_train_secs=1800,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HU1a1bf9k5sY"
      },
      "source": [
        "### Remote training\n",
        "\n",
        "Now, train a TabNet tabular classification model as a remote training job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hk3Gp95blJ4K"
      },
      "outputs": [],
      "source": [
        "REMOTE_JOB_NAME = \"test-sdk2-remote-tabnet-training\"\n",
        "REMOTE_JOB_BUCKET = \"/\".join([BUCKET_URI, REMOTE_JOB_NAME])\n",
        "\n",
        "# Sets the staging bucket for remote training\n",
        "tabnet_trainer.fit.vertex.remote_config.staging_bucket = (\n",
        "    REMOTE_JOB_BUCKET + f\"_{uuid.uuid4()}\"\n",
        ")\n",
        "\n",
        "# Executes remote training\n",
        "tabnet_trainer.fit(training_data=training_data, validation_data=validation_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyoVwMoPlYUi"
      },
      "source": [
        "### Local prediction\n",
        "\n",
        "Finally, make a local prediction with the trained tabular classification model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c_3rQyLeleig"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "predictions = tabnet_trainer.predict(eval_data)\n",
        "accuracy_score(y_test, predictions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tXCf6wCHwvAt"
      },
      "source": [
        "## Tabular regression model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HtTBSRnu0RGX"
      },
      "source": [
        "### Initialize TabNetTrainer\n",
        "\n",
        "Next, initialize the TabNetTrainer for training a tabular regression model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-8hp__390V__"
      },
      "outputs": [],
      "source": [
        "tabnet_trainer = TabNetTrainer(\n",
        "    model_type=\"regression\",\n",
        "    target_column=\"target\",\n",
        "    learning_rate=0.01,\n",
        "    max_train_secs=1800,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3CDcyfw0g9K"
      },
      "source": [
        "### Remote training\n",
        "\n",
        "Now, train a TabNet tabular regression model as a remote training job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I3OgDbb405N1"
      },
      "outputs": [],
      "source": [
        "# Sets the staging bucket for remote training\n",
        "tabnet_trainer.fit.vertex.remote_config.staging_bucket = (\n",
        "    REMOTE_JOB_BUCKET + f\"_{uuid.uuid4()}\"\n",
        ")\n",
        "\n",
        "# Executes remote training\n",
        "tabnet_trainer.fit(training_data=training_data, validation_data=validation_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyoVwMoPlYUi"
      },
      "source": [
        "### Local prediction\n",
        "\n",
        "Finally, make a local prediction with the trained tabular regression model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x_y_cwbx0-0O"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import mean_squared_error\n",
        "\n",
        "predictions = tabnet_trainer.predict(eval_data)\n",
        "mean_squared_error(y_test, predictions, squared=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6944380b1e25"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "655e39c67585"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "delete_bucket = False\n",
        "\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil rm -rf {BUCKET_URI}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "sdk2_remote_tabnet_training.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}