notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Delete Outdated Experiments in Vertex AI TensorBoard\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fexperiments%2Fdelete_outdated_tensorboard_experiments.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/vertex-ai-samples/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24743cf4a1e1"
      },
      "source": [
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Vertex AI has a new Vertex AI TensorBoard billing model. Charges are based on monthly storage at a cost of  $10/GB. It’s therefore prudent to delete old Vertex AI TensorBoard Experiments to avoid unnecessary storage costs.\n",
        "\n",
        "Learn more about [Vertex AI TensorBoard](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to delete outdated TensorBoard Experiments to avoid unnecessary storage costs.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI TensorBoard\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- How to delete the TB Experiment with a predefined key-value label pair `<label_key, label_value>`\n",
        "\n",
        "- How to delete the TB Experiments created before the  `create_time`\n",
        "\n",
        "- How to delete the TB Experiments created before the  `update_time`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "No dataset is used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1ea81ac77f0"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    USER = \"--user\"\n",
        "else:\n",
        "    USER = \"\"\n",
        "! pip3 install {USER} --upgrade google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the ### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab.following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c87a2a5d7e35"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dccb1c8feb6"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc7251520a07"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c2fc3d7b6bfa"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "45769dc0c97d"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "\n",
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NUJOpq7_-nt6"
      },
      "source": [
        "### Define constants\n",
        "\n",
        "Define variables you use in this tutorial. In particular, you set\n",
        "\n",
        "- `CREATE_TIME_CUT` : delete TensorBoard experiments that were created before CREATE_TIME_CUT. For example, `2022-12-31`.\n",
        "\n",
        "- `UPDATE_TIME_CUT` : delete TensorBoard experiments that were created before UPDATE_TIME_CUT. For example, `2022-12-31`.\n",
        "\n",
        "- `DETAILED_LOG` : a booled variable to see TensorBoard deletion progress. If True, it shows progress by experiments. Otherwise, it reports progress per 100 experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0AkyoMre-qe5"
      },
      "outputs": [],
      "source": [
        "CREATE_TIME_CUT = \"\"  # @param {type:\"string\"}\n",
        "\n",
        "UPDATE_TIME_CUT = \"\"  # @param {type:\"string\"}\n",
        "\n",
        "DETAILED_LOG = True  # @param {type: \"boolean\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n68J5KJ4ERbF"
      },
      "source": [
        "### Define Helpers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JIXC-tVrETET"
      },
      "outputs": [],
      "source": [
        "def clean_up_by_label(tensorboard_instance, label_key, label_value):\n",
        "    \"\"\"Delete the TB Experiment with the key-value label pair <label_key, label_value>\"\"\"\n",
        "    # List tensorboard experiments\n",
        "    tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
        "        tensorboard_name=tensorboard_instance.resource_name\n",
        "    )\n",
        "    # Get the number of tensorboard experiments\n",
        "    num_tensorboard_experiments = len(tensorboard_experiments)\n",
        "    # For each experiment\n",
        "    for i in range(num_tensorboard_experiments):\n",
        "        tensorboard_experiment = tensorboard_experiments[i]\n",
        "        if DETAILED_LOG or (i % 100 == 0):\n",
        "            print(\n",
        "                f\">>>checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
        "            )\n",
        "        # Get experiment labels\n",
        "        labels = tensorboard_experiment.labels\n",
        "        # Filter by label\n",
        "        if label_key in labels and labels[label_key] == label_value:\n",
        "            # Delete experiment\n",
        "            tensorboard_experiment.delete()\n",
        "\n",
        "\n",
        "def clean_up_by_create_time(tensorboard_instance, create_time_cut):\n",
        "    \"\"\"Delete the TB Experiment with the `create_time`\"\"\"\n",
        "    # List tensorboard experiments\n",
        "    tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
        "        tensorboard_name=tensorboard_instance.resource_name, order_by=\"create_time\"\n",
        "    )\n",
        "    # Get the number of tensorboard experiments\n",
        "    num_tensorboard_experiments = len(tensorboard_experiments)\n",
        "    # For each experiment\n",
        "    for i in range(num_tensorboard_experiments):\n",
        "        tensorboard_experiment = tensorboard_experiments[i]\n",
        "        if DETAILED_LOG or (i % 100 == 0):\n",
        "            print(\n",
        "                f\">>> checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
        "            )\n",
        "        # Filter by create_time\n",
        "        if str(tensorboard_experiment.create_time) < create_time_cut:\n",
        "            # Delete experiment\n",
        "            tensorboard_experiment.delete()\n",
        "        else:\n",
        "            break\n",
        "\n",
        "\n",
        "def clean_up_by_update_time(tensorboard_instance, update_time_cut):\n",
        "    \"\"\"Delete the TB Experiment with with the `update_time`\"\"\"\n",
        "    # List tensorboard experiments\n",
        "    tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
        "        tensorboard_name=tensorboard_instance.resource_name, order_by=\"update_time\"\n",
        "    )\n",
        "    # Get the number of tensorboard experiments\n",
        "    num_tensorboard_experiments = len(tensorboard_experiments)\n",
        "    # For each experiment\n",
        "    for i in range(num_tensorboard_experiments):\n",
        "        tensorboard_experiment = tensorboard_experiments[i]\n",
        "        if DETAILED_LOG or (i % 100 == 0):\n",
        "            print(\n",
        "                f\">>> checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
        "            )\n",
        "        # Filter by update_time\n",
        "        if str(tensorboard_experiment.update_time) < update_time_cut:\n",
        "            tensorboard_experiment.delete()\n",
        "        else:\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UCUOlTLYCT0B"
      },
      "source": [
        "## Delete outdated Vertex AI TensorBoard Experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e4672ef33691"
      },
      "source": [
        "### Set delete outdated TensorBoard experiments \n",
        "\n",
        "Initialize a flag variable to start deleting outdated TensorBoard experiments and a flag variable to choose the deleting method. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8cdc326c9823"
      },
      "outputs": [],
      "source": [
        "delete_tb_experiments = False  # @param {type: \"boolean\"}\n",
        "\n",
        "delete_method = \"\"  # @param [\"by_label\", \"by_create_time\", \"by_update_time\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjJVARe5V-MG"
      },
      "source": [
        "### Delete a TensorBoard instance\n",
        "\n",
        "To delete a TensorBoard instance, you need `TENSORBOARD_INSTANCE` ID which uniquely identifies the TensorBoard instance where you run experiments.\n",
        "\n",
        "To get the TensorBoard instance ID, you can either\n",
        "\n",
        "- go to the cloud console UI, Vertex AI > Experiments > TensorBoard Instances, or\n",
        "- use the list command below to list all TensorBoard instances for your project and region.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eqM2pIO8_4Y7"
      },
      "outputs": [],
      "source": [
        "tensorboard_instances = aiplatform.Tensorboard.list(\n",
        "    project=PROJECT_ID, location=LOCATION\n",
        ")\n",
        "print(tensorboard_instances)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wT_uPArACMPX"
      },
      "source": [
        "Set the tensorboard instance id for which you want to delete experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hUync8E9CLcm"
      },
      "outputs": [],
      "source": [
        "if delete_tb_experiments:\n",
        "\n",
        "    TENSORBOARD_INSTANCE_ID = \"\"  # @param {type:\"string\"}\n",
        "\n",
        "    TENSORBOARD_INSTANCE = aiplatform.Tensorboard(\n",
        "        project=PROJECT_ID, location=LOCATION, tensorboard_name=TENSORBOARD_INSTANCE_ID\n",
        "    )\n",
        "    print(TENSORBOARD_INSTANCE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OOp22fxdHqmT"
      },
      "source": [
        "### Delete a TensorBoard Experiment using the key-value label pair\n",
        "\n",
        "You delete a TensorBoard experiment using a predefined `label_key` and `label_value`.\n",
        "For example, you may have assigned `delete` label key and `true` label value to indicate all the TensorBoard experiments you want to delete.\n",
        "TensorBoard experiment labels can be set at creation or using the API update method.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N7KajCsjij1g"
      },
      "outputs": [],
      "source": [
        "LABEL_KEY = \"delete\"  # @param {type:\"string\"}\n",
        "LABEL_VALUE = \"true\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N45samDzWBO2"
      },
      "outputs": [],
      "source": [
        "if delete_tb_experiments and delete_method == \"by_label\":\n",
        "    clean_up_by_label(TENSORBOARD_INSTANCE, LABEL_KEY, LABEL_VALUE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrEVqlgSIL6K"
      },
      "source": [
        "### Delete a TensorBoard Experiment with `create_time`\n",
        "\n",
        "Delete a TensorBoard experiment using `create_time` field"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4_bjNeQ7p87A"
      },
      "outputs": [],
      "source": [
        "if delete_tb_experiments and delete_method == \"by_create_time\":\n",
        "    clean_up_by_create_time(TENSORBOARD_INSTANCE, CREATE_TIME_CUT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGNlSbIrIP_o"
      },
      "source": [
        "### Delete a TensorBoard Experiment with `update_time`\n",
        "\n",
        "Delete a TensorBoard Experiment using a predefined `update_time` field"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SDyc_a8XwEve"
      },
      "outputs": [],
      "source": [
        "if delete_tb_experiments and delete_method == \"by_update_time\":\n",
        "    clean_up_by_update_time(TENSORBOARD_INSTANCE, UPDATE_TIME_CUT)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "delete_outdated_tensorboard_experiments.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}