notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic"
      },
      "source": [
        "# Vertex AI Pipelines:  model upload, predict, and evaluate using google-cloud-pipeline-components\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\" target='_blank'>\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "     </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>\n",
        "*Note: This notebook uses KFP 1.x and GCPC 1.x. We recommend using 2.x*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:pipelines,custom"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to use the components defined in [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) in conjunction with an experimental `evaluation` method, to build a [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) workflow that uploads a tabular custom model as a `Model` resource, creates a `BatchPredictionJob` resource, and evaluates the `Model` resource with the `BatchPredictionJob` results to create an evaluation `system.Metrics` artifact. \n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [Vertex AI Model components](https://cloud.google.com/vertex-ai/docs/pipelines/model-endpoint-component)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:pipelines,custom"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to evaluate a custom model using a pipeline with components from `google_cloud_pipeline_components` and a custom pipeline component you build.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Pipelines\n",
        "- Vertex AI Model Registry\n",
        "- Vertex AI Batch Prediction\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Upload a pre-trained model as a `Model` resource.\n",
        "- Run a `BatchPredictionJob` on the `Model` resource with ground truth data.\n",
        "- Generate evaluation `Metrics` artifact about the `Model` resource.\n",
        "- Compare the evaluation metrics to a threshold.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:bikes_weather,lrg"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is part of the [safe driver prediction Kaggle competition](https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/overview). The model has been trained on this data, and ground truth is used for evaluation.\n",
        "\n",
        "The dataset predicts the whether or not a claim was filed for the policy holder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QTmIIQ8QBfb6"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
        "                                 'google-cloud-pipeline-components<2' \\\n",
        "                                 'kfp<2'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "3. [Enable the Vertex AI API]\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### UUID\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MIe7f62gBfb_"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of a specifed length(default=8)\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
        "\n",
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated.\n",
        "\n",
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ce6043da7b33"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0367eac06a10"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "21ad4dbb4a61"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c13224697bfb"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets.\n",
        "\n",
        "- *{Note to notebook author: For any user-provided strings that need to be unique (like bucket names or model ID's), append \"-unique\" to the end so proper testing can occur}*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Service Account\n",
        "\n",
        "You use a service account to create Vertex AI Pipeline jobs.\n",
        "\n",
        "If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    else:  # IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import kfp\n",
        "from google.cloud import aiplatform\n",
        "from google_cloud_pipeline_components.experimental.evaluation import \\\n",
        "    ModelEvaluationOp as evaluation_op\n",
        "from google_cloud_pipeline_components.types import artifact_types\n",
        "from google_cloud_pipeline_components.v1.batch_predict_job import \\\n",
        "    ModelBatchPredictOp as batch_prediction_op\n",
        "from google_cloud_pipeline_components.v1.model import \\\n",
        "    ModelUploadOp as model_upload_op\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.components import importer_node\n",
        "from kfp.v2.dsl import Input, Metrics, component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pipeline_constants"
      },
      "source": [
        "Setup up the following constant for Vertex AI Pipelines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Uw-NxCiBfcD"
      },
      "outputs": [],
      "source": [
        "PIPELINE_ROOT = \"{}/pipeline_root/safe_driver\".format(BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m3atE5jVBfcE"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI, location=REGION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7a5cc8c7f3c"
      },
      "source": [
        "## Create component for comparing evalution metrics to a threshold\n",
        "\n",
        "First, you create your own component that takes the evaluation metrics artifact as input, checks the threshold and return yes/no decision. It is used in a subsequent dsl.Condition() to decide whether the model should proceed to the next step i.e., online deployment.\n",
        "\n",
        "The component takes the following parameters:\n",
        "\n",
        "- `eval_metrics`: The evaluation metrics artifact returned from `ModelEvaluation` component.\n",
        "- `metric_name`: The key name for the metric entry to make the comparison to.\n",
        "- `threshold`: The threshold for the metric value for a yes/no decision."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5cf109a9bcca"
      },
      "outputs": [],
      "source": [
        "@component()\n",
        "def compare(eval_metrics: Input[Metrics], metric_name: str, threshold: float) -> str:\n",
        "    path = eval_metrics.path\n",
        "    # print(\"PATH\", path)\n",
        "\n",
        "    gs_prefix = \"gs://\"\n",
        "    gcsfuse_prefix = \"/gcs/\"\n",
        "    if path.startswith(gs_prefix):\n",
        "        path = path.replace(gs_prefix, gcsfuse_prefix)\n",
        "\n",
        "    import json\n",
        "\n",
        "    with open(path, \"r\") as f:\n",
        "        data = json.load(f)\n",
        "\n",
        "    slices = data[\"slicedMetrics\"]\n",
        "\n",
        "    metrics = slices[0][\"metrics\"][\"classification\"]\n",
        "    # print(\"METRIC KEYS\", metrics.keys())\n",
        "\n",
        "    value = metrics[metric_name]\n",
        "    if value > threshold:\n",
        "        return \"true\"\n",
        "\n",
        "    return \"false\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_pipeline:gcpc,bikes_weather,lrg"
      },
      "source": [
        "## Define evaluation pipeline that uses components from `google_cloud_pipeline_components`\n",
        "\n",
        "Next, define the pipeline.\n",
        "\n",
        "[`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) components used to define the pipeline are: upload the model, run batch prediction, and evaluate the model with the given predictions.\n",
        "\n",
        "View the definition of the [upload model component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/aiplatform/model/upload_model/component.yaml).\n",
        "\n",
        "View the definition of the [batch prediction component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/aiplatform/batch_predict_job/component.yaml).\n",
        "\n",
        "View the definition of the [experimental evaluation component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/experimental/evaluation/component.yaml)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NvTkSfOeBfcF"
      },
      "outputs": [],
      "source": [
        "DATA_URIS = [\n",
        "    \"gs://cloud-samples-data/vertex-ai/dataset-management/datasets/safe_driver/dataset_safe_driver_train_10k.csv\"\n",
        "]\n",
        "MODEL_URI = \"gs://cloud-samples-data/vertex-ai/google-cloud-aiplatform-ci-artifacts/models/safe_driver/model\"\n",
        "# Create working dir\n",
        "WORKING_DIR = f\"{PIPELINE_ROOT}/{UUID}\"\n",
        "MODEL_DISPLAY_NAME = f\"safe-driver-{UUID}\"\n",
        "BATCH_PREDICTION_DISPLAY_NAME = f\"batch-prediction-on-pipelines-model-{UUID}\"\n",
        "\n",
        "\n",
        "@kfp.dsl.pipeline(name=\"upload-evaluate-\" + UUID)\n",
        "def pipeline(\n",
        "    metric: str,\n",
        "    threshold: float,\n",
        "    project: str = PROJECT_ID,\n",
        "    model_display_name: str = MODEL_DISPLAY_NAME,\n",
        "    batch_prediction_display_name: str = BATCH_PREDICTION_DISPLAY_NAME,\n",
        "    batch_prediction_data_uris: list = DATA_URIS,\n",
        "):\n",
        "    import_unmanaged_model_task = importer_node.importer(\n",
        "        artifact_uri=MODEL_URI,\n",
        "        artifact_class=artifact_types.UnmanagedContainerModel,\n",
        "        metadata={\n",
        "            \"artifactUri\": MODEL_URI,\n",
        "            \"predictSchemata\": {\n",
        "                \"predictionSchemaUri\": MODEL_URI + \"/prediction_schema.yaml\",\n",
        "                \"instanceSchemaUri\": MODEL_URI + \"/instance.yaml\",\n",
        "            },\n",
        "            \"containerSpec\": {\n",
        "                \"imageUri\": \"us-docker.pkg.dev/vertex-ai/automl-tabular/prediction-server:prod\",\n",
        "                \"healthRoute\": \"/health\",\n",
        "                \"predictRoute\": \"/predict\",\n",
        "            },\n",
        "        },\n",
        "    )\n",
        "\n",
        "    model_task = model_upload_op(\n",
        "        project=project,\n",
        "        display_name=model_display_name,\n",
        "        unmanaged_container_model=import_unmanaged_model_task.outputs[\"artifact\"],\n",
        "    )\n",
        "\n",
        "    batch_prediction_task = batch_prediction_op(\n",
        "        project=project,\n",
        "        model=model_task.outputs[\"model\"],\n",
        "        job_display_name=batch_prediction_display_name,\n",
        "        gcs_source_uris=batch_prediction_data_uris,\n",
        "        instances_format=\"csv\",\n",
        "        predictions_format=\"jsonl\",\n",
        "        gcs_destination_output_uri_prefix=WORKING_DIR,\n",
        "        machine_type=\"n1-standard-32\",\n",
        "    )\n",
        "\n",
        "    eval_task = evaluation_op(\n",
        "        project=project,\n",
        "        root_dir=WORKING_DIR,\n",
        "        problem_type=\"classification\",\n",
        "        classification_type=\"multiclass\",\n",
        "        ground_truth_column=\"target\",\n",
        "        class_names=[\"0\", \"1\"],\n",
        "        predictions_format=\"jsonl\",\n",
        "        batch_prediction_job=batch_prediction_task.outputs[\"batchpredictionjob\"],\n",
        "    )\n",
        "\n",
        "    _ = compare(\n",
        "        eval_metrics=eval_task.outputs[\"evaluation_metrics\"],\n",
        "        metric_name=metric,\n",
        "        threshold=threshold,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compile_pipeline"
      },
      "source": [
        "## Compile the pipeline\n",
        "\n",
        "Next, compile the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N7kt8MqfBfcG"
      },
      "outputs": [],
      "source": [
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline,\n",
        "    package_path=\"evaluation_demo_pipeline.json\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline:custom"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "Next, run the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u5s7gMoABfcG"
      },
      "outputs": [],
      "source": [
        "DISPLAY_NAME = \"safe_driver\" + UUID\n",
        "\n",
        "job = aiplatform.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=\"evaluation_demo_pipeline.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\"metric\": \"auPrc\", \"threshold\": 0.95},\n",
        "    enable_caching=True,\n",
        ")\n",
        "\n",
        "job.run()\n",
        "\n",
        "! rm evaluation_demo_pipeline.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_run:custom"
      },
      "source": [
        "Click on the generated link to see your run in the Cloud Console.\n",
        "\n",
        "In the UI, the nodes of pipeline DAG expand or collapse when you click on them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipleline_results:bqml"
      },
      "source": [
        "### View the pipeline results and evaluation metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "view_pipleline_results:bqml"
      },
      "outputs": [],
      "source": [
        "PROJECT_NUMBER = job.gca_resource.name.split(\"/\")[1]\n",
        "print(PROJECT_NUMBER)\n",
        "\n",
        "\n",
        "def print_pipeline_output(job, output_task_name):\n",
        "    JOB_ID = job.name\n",
        "    print(JOB_ID)\n",
        "    artifact = \"\"\n",
        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
        "        EXECUTE_OUTPUT = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/executor_output.json\"\n",
        "        )\n",
        "        GCP_RESOURCES = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/gcp_resources\"\n",
        "        )\n",
        "        EVALUATION_METRICS = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/evaluation_metrics\"\n",
        "        )\n",
        "        # Check if file exists, 0 is success\n",
        "        !gsutil -q stat $EXECUTE_OUTPUT\n",
        "        if _exit_code == 0:\n",
        "            ! gsutil cat $EXECUTE_OUTPUT\n",
        "            artifact = EXECUTE_OUTPUT\n",
        "            break\n",
        "        !gsutil -q stat $GCP_RESOURCES\n",
        "        if _exit_code == 0:\n",
        "            ! gsutil cat $GCP_RESOURCES\n",
        "            artifact = GCP_RESOURCES\n",
        "            break\n",
        "        !gsutil -q stat $EVALUATION_METRICS\n",
        "        if _exit_code == 0:\n",
        "            ! gsutil cat $EVALUATION_METRICS\n",
        "            artifact = EVALUATION_METRICS\n",
        "            break\n",
        "\n",
        "    return artifact\n",
        "\n",
        "\n",
        "print(\"model-upload\")\n",
        "artifacts = print_pipeline_output(job, \"model-upload\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-batch-predict\")\n",
        "artifacts = print_pipeline_output(job, \"model-batch-predict\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-evaluation\")\n",
        "metrics = print_pipeline_output(job, \"model-evaluation\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"compare\")\n",
        "artifacts = print_pipeline_output(job, \"compare\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "delete_pipeline"
      },
      "source": [
        "### Delete pipeline job\n",
        "\n",
        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "delete_pipeline"
      },
      "outputs": [],
      "source": [
        "job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial.\n",
        "\n",
        "- Model\n",
        "- Batch Job\n",
        "- Cloud Storage Bucket (Set `delete_bucket` to **True** to delete the Cloud Storage bucket)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JgCXD3X-BfcG"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "delete_bucket = False\n",
        "\n",
        "# Delete the created model\n",
        "models = aiplatform.Model.list(\n",
        "    filter=f\"display_name={MODEL_DISPLAY_NAME}\", order_by=\"create_time\"\n",
        ")\n",
        "if len(models) > 0:\n",
        "    model = models[0]\n",
        "    model.delete()\n",
        "    print(\"Deleted model:\", model)\n",
        "\n",
        "# Delete the created batch-prediction job\n",
        "batch_predictions = aiplatform.BatchPredictionJob.list(\n",
        "    filter=f\"display_name={BATCH_PREDICTION_DISPLAY_NAME}\",\n",
        "    order_by=\"create_time\",\n",
        ")\n",
        "if len(batch_predictions) > 0:\n",
        "    batch_prediction = batch_predictions[0]\n",
        "    batch_prediction.delete()\n",
        "    print(\"Deleted batch prediction job:\", batch_prediction)\n",
        "\n",
        "# Delete the Cloud Storage bucket\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}