notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Get started with importing a custom model evaluation to the Vertex AI Model Registry\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_evaluation%2Fget_started_with_custom_model_evaluation_import.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24743cf4a1e1"
      },
      "source": [
        "**_NOTE_**: This notebook is tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use Vertex AI Model Evaluation to import a custom model evaluation to an existing Vertex AI Model Registry entry.\n",
        "\n",
        "Learn more about [Model evaluation in Vertex AI](https://cloud.google.com/vertex-ai/docs/evaluation/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to construct and upload a custom model evaluation, and upload the custom model evaluation to a model resource entry in Vertex AI Model Registry.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Model Evaluation\n",
        "- Vertex AI Model Registry\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Import a pretrained (blessed) model to the Vertex AI Model Registry.\n",
        "- Construct a custom model evaluation.\n",
        "- Import the model evaluation metrics to the corresponding model in the Vertex AI Model Registry.\n",
        "- List the model evaluation for the corresponding model in the Vertex AI Model Registry.\n",
        "- Construct a second custom model evaluation.\n",
        "- Import the second model evaluation metrics to the corresponding model in the Vertex AI Model Registry.\n",
        "- List the second model evaluation for the corresponding model in the Vertex AI Model Registry.\n",
        "\n",
        "Learn more about [Model Evaluation in Vertex AI](https://cloud.google.com/vertex-ai/docs/evaluation/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Model\n",
        "\n",
        "This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.\n",
        "\n",
        "Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "USER=''\n",
        "! pip3 install {USER} --upgrade google-cloud-aiplatform \\\n",
        "                                tensorflow==2.15.1 \\\n",
        "                                tensorflow-hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets.\n",
        "\n",
        "- *{Note to notebook author: For any user-provided strings that need to be unique (like bucket names or model ID's), append \"-unique\" to the end so proper testing can occur}*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from google.cloud import aiplatform\n",
        "from google.cloud.aiplatform import gapic"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aip.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators), and [GPU pricing](https://cloud.google.com/compute/gpus-pricing)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for prediction.\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2.15.1\".replace(\".\", \"-\")\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from TensorFlow Hub\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a Vertex AI model resource. Once you have a Vertex AI model resource, the model can be deployed to a Vertex AI endpoint resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "tfhub_model = tf.keras.Sequential(\n",
        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
        ")\n",
        "\n",
        "tfhub_model.build([None, 32, 32, 3])\n",
        "\n",
        "tfhub_model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64618c713db9"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
        "tfhub_model.save(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "### Upload the TensorFlow Hub model to a Vertex AI model resource\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model into a Vertex AI model resource using the `upload()` method, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the model resource.\n",
        "- `artifact_uri`: The Cloud Storage location of the model package.\n",
        "- `serving_container_image_uri`: The serving container image.\n",
        "\n",
        "Uploading a model to a Vertex AI model resource is a long running operation that may take a few moments \n",
        "\n",
        "*Note:* When you upload the model artifacts to a Vertex AI model resource, you specify the corresponding deployment container image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "model = aiplatform.Model.upload(\n",
        "    display_name=\"resnet\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    is_default_version=True,\n",
        "    version_aliases=[\"v1\"],\n",
        ")\n",
        "\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c11e98ef5391"
      },
      "source": [
        "## Introduction to custom model evaluations\n",
        "\n",
        "When training a custom model, you generally perform some form of evaluation of the trained model. Your custom model evaluation can then be imported to the corresponding model in the Vertex AI Model Registry using the `import_model_evaluation()` method. Once imported, you can retreive the custom model evaluation with the `list_model_evaluations()` method. \n",
        "\n",
        "The Vertex AI Model Registry supports importing multiple model evaluations for a model where each evaluation is distinquished by a unique `display_name`.\n",
        "\n",
        "\n",
        "### Create a model evaluation\n",
        "\n",
        "First, you create a model evaluation in a format that corresponds to one of the predefined schemas for model evaluations. In this example, you use the schema for a classification metric, and specify the following subset of evaluation metrics as a dictionary:\n",
        "\n",
        "- `logLoss`: The log loss.\n",
        "- `auPrc`: The accuracy.\n",
        "\n",
        "You then construct the `ModelEvaluation` object with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the evaluation metric.\n",
        "- `metrics_schema_uri`: The schema for the specific type of evaluation metrics.\n",
        "- `metrics`: The dictionary with the evaluation metrics.\n",
        "\n",
        "Learn more about [Schemas for evaluation metrics](https://cloud.google.com/vertex-ai/docs/evaluation/introduction#features)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e9af222db292"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.4, \"auPrc\": 0.85}\n",
        "print(metrics)\n",
        "\n",
        "model_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"eval\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68870bd8194d"
      },
      "source": [
        "### Upload the evaluation metrics to the Model Registry\n",
        "\n",
        "Next, upload the model's evaluation from the custom training job to the corresponding entry in the Vertex AI Model Registry.\n",
        "\n",
        "Currently, there isn't yet support for this method in the SDK. Instead, you use the lower level GAPIC API interface."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3044788848af"
      },
      "outputs": [],
      "source": [
        "API_ENDPOINT = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
        "client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "\n",
        "client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_eval)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "337319a19efe"
      },
      "source": [
        "### List the custom model evaluation\n",
        "\n",
        "Now that you have uploaded your custom evaluation metric to the corresponding model in the Vertex AI Model Registry, you can retrieve it using the `list_model_evaluations()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1b3374e988cf"
      },
      "outputs": [],
      "source": [
        "evaluation = model.list_model_evaluations()[0]\n",
        "print(evaluation.gca_resource)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68870bd8194d"
      },
      "source": [
        "### Upload a second evaluation metrics to the Model Registry\n",
        "\n",
        "Next, upload a second model evaluation to the corresponding entry in the Vertex AI Model Registry. In this example, we refer to first evaluation metric as `eval` (from training) and the second as `prod` (from production data)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9b864c0ba07f"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.2, \"auPrc\": 0.87}\n",
        "print(metrics)\n",
        "\n",
        "model_prod = gapic.ModelEvaluation(\n",
        "    display_name=\"prod\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")\n",
        "\n",
        "client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_prod)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "337319a19efe"
      },
      "source": [
        "### List a specific custom model evaluation\n",
        "\n",
        "Now that you have uploaded your second custom evaluation metric to the corresponding model in the Vertex AI Model Registry, you can retrieve a specific evaluation by filtering using `display_name`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1b3374e988cf"
      },
      "outputs": [],
      "source": [
        "evaluations = model.list_model_evaluations()\n",
        "for evaluation in evaluations:\n",
        "    if evaluation.display_name == \"prod\":\n",
        "        print(evaluation.gca_resource)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3a3993dc813"
      },
      "source": [
        "### Upload version 2 of the TFHub model to the Vertex AI Model Registry\n",
        "\n",
        "Next, you upload the second version of the TFHub model as a model resource in the Vertex AI Model Registry, with the additional following parameters:\n",
        "\n",
        "- `parent_model`: The resource name or model ID of an existing model that the newly-uploaded model is a version of. Only set this field when uploading a new version of an existing model.\n",
        "- `is_default_version`:  When set to `True`, the newly uploaded model version automatically includes the alias \"default\".\n",
        "- `version_ailiases`: User defined list of alternative alias names for the model version, such as `production`.\n",
        "- `version_description`: User description of the model version.\n",
        "\n",
        "When a subsequent model version is created in the Vertex AI Model Registry, the property `version_id` is automatically incremented. In this example, it's set to 2 (2nd version)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "852be5e5f964"
      },
      "outputs": [],
      "source": [
        "model_v2 = aiplatform.Model.upload(\n",
        "    display_name=\"resnet\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    parent_model=model.resource_name,\n",
        "    is_default_version=True,\n",
        "    version_aliases=[\"v2\"],\n",
        "    version_description=\"This is the second version of the model\",\n",
        ")\n",
        "\n",
        "print(model_v2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68870bd8194d"
      },
      "source": [
        "#### Upload an evaluation metrics for version 2 of the model to the Model Registry\n",
        "\n",
        "Next, upload a model evaluation to the corresponding model version in the Vertex AI Model Registry. *Note*, use `model_v2.resource_name` to refer to version 2 of this model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9b864c0ba07f"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.0, \"auPrc\": 0.91}\n",
        "print(metrics)\n",
        "\n",
        "model_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"eval\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")\n",
        "\n",
        "client.import_model_evaluation(\n",
        "    parent=model_v2.resource_name, model_evaluation=model_eval\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4e72b21b06b"
      },
      "source": [
        "### List the evaluations for both versions of the model\n",
        "\n",
        "Finally, list the number of evaluations for both versions of the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "369a23d392a4"
      },
      "outputs": [],
      "source": [
        "evaluations = model.list_model_evaluations()\n",
        "print(\"Model v1 no. of evaluations\", len(evaluations))\n",
        "evaluations = model_v2.list_model_evaluations()\n",
        "print(\"Model v2 no. of evaluations\", len(evaluations))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# Delete model resource\n",
        "model.delete()\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "delete_bucket = False  # set True for deletion\n",
        "if delete_bucket:\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "get_started_with_custom_model_evaluation_import.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}