examples/training/01_clustering_example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12b45b0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2024 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8790d7ed",
   "metadata": {},
   "source": [
    "# AutoMLOps - Clustering Example\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/automlops/blob/main/examples/training/01_clustering_example.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/01_clustering_example.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/main/examples/training/01_clustering_example.ipynb\">\n",
    "        <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>\n",
    "<br/><br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f938540",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "In this tutorial, you will build a [Vertex AI](https://cloud.google.com/vertex-ai) pipeline, complete with an integrated CI/CD pipeline. This tutorial will walk you through how to use AutoMLOps to define, create and run pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22881a6a",
   "metadata": {},
   "source": [
    "# Objective\n",
    "In this tutorial, you will learn how to create and run MLOps pipelines integrated with CI/CD. The example pipeline fits a K Means model to the Sklearn Iris dataset; the pipeline goes through a very basic workflow:\n",
    "1. create_dataset: A custom component that loads the sklearn Iris dataset and writes it to GCS.\n",
    "2. fit_kmeans: A custom component that determines K Means clusters within the data.\n",
    "\n",
    "# Prerequisites\n",
    "\n",
    "In order to use AutoMLOps, the following are required:\n",
    "\n",
    "- Python 3.7 - 3.10\n",
    "- [Google Cloud SDK 407.0.0](https://cloud.google.com/sdk/gcloud/reference)\n",
    "- [beta 2022.10.21](https://cloud.google.com/sdk/gcloud/reference/beta)\n",
    "- `git` installed\n",
    "- `git` logged-in:\n",
    "```\n",
    "  git config --global user.email \"you@example.com\"\n",
    "  git config --global user.name \"Your Name\"\n",
    "```\n",
    "- [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/provide-credentials-adc) are setup. This can be done through the following commands:\n",
    "```\n",
    "gcloud auth application-default login\n",
    "gcloud config set account <account@example.com>\n",
    "```\n",
    "\n",
    "# APIs & IAM\n",
    "Based on the user options selection, AutoMLOps will enable up to the following APIs during the provision step:\n",
    "- [aiplatform.googleapis.com](https://cloud.google.com/vertex-ai/docs/reference/rest)\n",
    "- [artifactregistry.googleapis.com](https://cloud.google.com/artifact-registry/docs/reference/rest)\n",
    "- [cloudbuild.googleapis.com](https://cloud.google.com/build/docs/api/reference/rest)\n",
    "- [cloudfunctions.googleapis.com](https://cloud.google.com/functions/docs/reference/rest)\n",
    "- [cloudresourcemanager.googleapis.com](https://cloud.google.com/resource-manager/reference/rest)\n",
    "- [cloudscheduler.googleapis.com](https://cloud.google.com/scheduler/docs/reference/rest)\n",
    "- [compute.googleapis.com](https://cloud.google.com/compute/docs/reference/rest/v1)\n",
    "- [iam.googleapis.com](https://cloud.google.com/iam/docs/reference/rest)\n",
    "- [iamcredentials.googleapis.com](https://cloud.google.com/iam/docs/reference/credentials/rest)\n",
    "- [logging.googleapis.com](https://cloud.google.com/logging/docs/reference/v2/rest)\n",
    "- [pubsub.googleapis.com](https://cloud.google.com/pubsub/docs/reference/rest)\n",
    "- [run.googleapis.com](https://cloud.google.com/run/docs/reference/rest)\n",
    "- [storage.googleapis.com](https://cloud.google.com/storage/docs/apis)\n",
    "- [sourcerepo.googleapis.com](https://cloud.google.com/source-repositories/docs/reference/rest)\n",
    "\n",
    "\n",
    "AutoMLOps will create the following service account and update [IAM permissions](https://cloud.google.com/iam/docs/understanding-roles) during the provision step:\n",
    "1. Pipeline Runner Service Account (defaults to: vertex-pipelines@PROJECT_ID.iam.gserviceaccount.com). Roles added:\n",
    "- roles/aiplatform.user\n",
    "- roles/artifactregistry.reader\n",
    "- roles/bigquery.user\n",
    "- roles/bigquery.dataEditor\n",
    "- roles/iam.serviceAccountUser\n",
    "- roles/storage.admin\n",
    "- roles/cloudfunctions.admin\n",
    "\n",
    "# User Guide\n",
    "\n",
    "For a user-guide, please view these [slides](../../AutoMLOps_User_Guide.pdf).\n",
    "\n",
    "# Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "- Vertex AI\n",
    "- Artifact Registry\n",
    "- Cloud Storage\n",
    "- Cloud Source Repository\n",
    "- Cloud Build\n",
    "- Cloud Run\n",
    "- Cloud Scheduler\n",
    "- Cloud Pub/Sub\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n",
    "\n",
    "# Ground-rules for using AutoMLOps\n",
    "1. Do not use variables, functions, code, etc. not defined within the scope of a custom component. These custom components will become containers and will have no reference to the out of scope code.\n",
    "2. Import statements and helper functions must be added inside the function. Provide parameter type hints.\n",
    "3. Test each of your components for accuracy and correctness before running them using AutoMLOps. We cannot fix bugs automatically; bugs are much more difficult to fix once they are made into pipelines.\n",
    "4. If you are using Kubeflow, be sure to define all the requirements needed to run the custom component - it can be easy to leave out packages which will cause the container to fail when running within a pipeline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a231b629",
   "metadata": {},
   "source": [
    "# Setup Git\n",
    "Set up your git configuration below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92f90b44",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git config --global user.email 'you@example.com'\n",
    "!git config --global user.name 'Your Name'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e4d190",
   "metadata": {},
   "source": [
    "# Install AutoMLOps\n",
    "\n",
    "Install AutoMLOps from [PyPI](https://pypi.org/project/google-cloud-automlops/), or locally by cloning the repo and running `pip install .`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94451868",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install google-cloud-automlops --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17db55d5",
   "metadata": {},
   "source": [
    "# Restart the kernel\n",
    "Once you've installed the AutoMLOps package, you need to restart the notebook kernel so it can find the package.\n",
    "\n",
    "**Note: Once this cell has finished running, continue on. You do not need to re-run any of the cells above.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57c53b86",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv('IS_TESTING'):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "250d511b",
   "metadata": {},
   "source": [
    "# Set your project ID\n",
    "Set your project ID below. If you don't know your project ID, leave the field blank and the following cells may be able to find it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "931ff517",
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = '[your-project-id]'  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "be0be295",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID: automlops-sandbox\n"
     ]
    }
   ],
   "source": [
    "if PROJECT_ID == '' or PROJECT_ID is None or PROJECT_ID == '[your-project-id]':\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print('Project ID:', PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2c36482d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5da8824",
   "metadata": {},
   "source": [
    "Set your Model_ID below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9e1f084d",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_ID = 'iris-k-means'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3fe994",
   "metadata": {},
   "source": [
    "# 1. AutoMLOps Clustering Example\n",
    "This workflow will define and generate a pipeline using AutoMLOps. AutoMLOps provides 2 functions for defining MLOps pipelines:\n",
    "\n",
    "- `AutoMLOps.component(...)`: Defines a component, which is a containerized python function.\n",
    "- `AutoMLOps.pipeline(...)`: Defines a pipeline, which is a series of components.\n",
    "\n",
    "AutoMLOps provides 6 functions for building and maintaining MLOps pipelines:\n",
    "\n",
    "- `AutoMLOps.generate(...)`: Generates the MLOps codebase. Users can specify the tooling and technologies they would like to use in their MLOps pipeline.\n",
    "- `AutoMLOps.provision(...)`: Runs provisioning scripts to create and maintain necessary infra for MLOps.\n",
    "- `AutoMLOps.deprovision(...)`: Runs deprovisioning scripts to tear down MLOps infra created using AutoMLOps.\n",
    "- `AutoMLOps.deploy(...)`: Builds and pushes component container, then triggers the pipeline job.\n",
    "- `AutoMLOps.launchAll(...)`: Runs `generate()`, `provision()`, and `deploy()` all in succession.\n",
    "- `AutoMLOps.monitor(...)`: Creates model monitoring jobs on deployed endpoints.\n",
    "\n",
    "Please see the [readme](https://github.com/GoogleCloudPlatform/automlops/blob/main/README.md) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2219ee67",
   "metadata": {},
   "source": [
    "## Import AutoMLOps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cd5ac7db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google_cloud_automlops import AutoMLOps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac0c712-acdd-4830-a9bd-44f41eaa590c",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "Define a custom component for loading and creating a dataset using `@AutoMLOps.component`. Import statements and helper functions must be added inside the function. Provide parameter type hints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "af35e8d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.component\n",
    "def create_dataset(data_path: str):\n",
    "    \"\"\"Custom component that loads the sklearn Iris dataset and writes it to GCS.\n",
    "\n",
    "    Args:\n",
    "        data_path: The gcs location to write the Iris data.\n",
    "    \"\"\"\n",
    "    import pandas as pd\n",
    "    from sklearn import datasets\n",
    "\n",
    "    # Load data\n",
    "    iris = datasets.load_iris()\n",
    "    data = pd.DataFrame(data=iris.data, columns=iris.feature_names)  \n",
    "    target = pd.DataFrame(data=iris.target, columns=['Species'])\n",
    "    df = pd.concat([data, target], axis=1)\n",
    "\n",
    "    # Calculate petal and sepal area and save dataset\n",
    "    df['sepal_area'] = df['sepal length (cm)'] * df['sepal width (cm)']\n",
    "    df['petal_area'] = df['petal length (cm)'] * df['petal width (cm)']\n",
    "    df.to_csv(data_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15a970b0",
   "metadata": {},
   "source": [
    "## Model Fitting\n",
    "Define a custom component for fitting KMeans clusters using `@AutoMLOps.component`. Import statements and helper functions must be added inside the function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b51b2f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.component\n",
    "def fit_kmeans(\n",
    "    data_path: str,\n",
    "    cluster_path: str\n",
    "):\n",
    "    \"\"\"Custom component that determines KMeans clusters.\n",
    "\n",
    "    Args:\n",
    "        data_path (str): The gcs location of the Iris data.\n",
    "        cluster_path (str): The gcs location of the Iris dataset augmented with clusters.\n",
    "    \"\"\"\n",
    "    from sklearn.cluster import KMeans\n",
    "    import pandas as pd\n",
    "\n",
    "    # Load data\n",
    "    df = pd.read_csv(data_path)\n",
    "\n",
    "    # Fit KMeans with 3 clusters to the sepal and petal area\n",
    "    kmeans = KMeans(n_clusters=3, n_init='auto')\n",
    "    df['Cluster'] = kmeans.fit_predict(df[['sepal_area', 'petal_area']])\n",
    "\n",
    "    df[['sepal_area', 'petal_area', 'Species', 'Cluster']].to_csv(cluster_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73d96dcb-a020-4bab-b0e3-1e32f6b2aecf",
   "metadata": {},
   "source": [
    "## Define the Pipeline\n",
    "Define your pipeline using `@AutoMLOps.pipeline`. You can optionally give the pipeline a name and description. Define the structure by listing the components to be called in your pipeline; use `.after` to specify the order of execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a36e0154",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.pipeline #(name='automlops-pipeline', description='This is an optional description')\n",
    "def pipeline(data_path: str,\n",
    "             cluster_path: str):\n",
    "\n",
    "    create_dataset_task = create_dataset(\n",
    "        data_path=data_path)\n",
    "\n",
    "    fit_kmeans_task = fit_kmeans(\n",
    "        data_path=data_path,\n",
    "        cluster_path=cluster_path).after(create_dataset_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3874f4f6",
   "metadata": {},
   "source": [
    "## Define the Pipeline Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4cb3786b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "date_bucket = datetime.datetime.now()\n",
    "pipeline_params = {\n",
    "    'data_path': f'gs://{PROJECT_ID}-bucket/kmeans/{date_bucket}/iris.csv',\n",
    "    'cluster_path': f'gs://{PROJECT_ID}-bucket/kmeans/{date_bucket}/iris_clusters.csv',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7e015f8",
   "metadata": {},
   "source": [
    "## Generate and Run the pipeline\n",
    "`AutoMLOps.generate(...)` generates the MLOps codebase. Users can specify the tooling and technologies they would like to use in their MLOps pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "73a48ac7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing directories under AutoMLOps/\n",
      "Writing configurations to AutoMLOps/configs/defaults.yaml\n",
      "Writing Kubeflow Pipelines code to AutoMLOps/pipelines, AutoMLOps/components, AutoMLOps/services\n",
      "Writing README.md to AutoMLOps/README.md\n",
      "Writing scripts to AutoMLOps/scripts\n",
      "Writing CloudBuild config to AutoMLOps/cloudbuild.yaml\n",
      "Code Generation Complete.\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.generate(project_id=PROJECT_ID,\n",
    "                   pipeline_params=pipeline_params,\n",
    "                   use_ci=True,\n",
    "                   naming_prefix=MODEL_ID,\n",
    "                   schedule_pattern='59 11 * * 0' # retrain every Sunday at Midnight\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc1d2c2c",
   "metadata": {},
   "source": [
    "`AutoMLOps.provision(...)` runs provisioning scripts to create and maintain necessary infra for MLOps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e9d3c0be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32m Setting up API services in project automlops-sandbox \u001b[0m\n",
      "Operation \"operations/acat.p2-45373616427-129a5ec2-c1b1-4af8-b33b-89f236eeb7ae\" finished successfully.\n",
      "\u001b[0;32m Setting up Artifact Registry in project automlops-sandbox \u001b[0m\n",
      "Listing items under project automlops-sandbox, location us-central1.\n",
      "\n",
      "Creating Artifact Registry: iris-k-means-artifact-registry in project automlops-sandbox\n",
      "Create request issued for: [iris-k-means-artifact-registry]\n",
      "Waiting for operation [projects/automlops-sandbox/locations/us-central1/operations/70548b29-cf18-4377-9263-e287eff77b10] to complete...\n",
      "......done.\n",
      "Created repository [iris-k-means-artifact-registry].\n",
      "\u001b[0;32m Setting up Storage Bucket in project automlops-sandbox \u001b[0m\n",
      "BucketNotFoundException: 404 gs://automlops-sandbox-iris-k-means-bucket bucket does not exist.\n",
      "Creating GS Bucket: automlops-sandbox-iris-k-means-bucket in project automlops-sandbox\n",
      "Creating gs://automlops-sandbox-iris-k-means-bucket/...\n",
      "\u001b[0;32m Setting up Pipeline Job Runner Service Account in project automlops-sandbox \u001b[0m\n",
      "Pipeline Runner Service Account         vertex-pipelines@automlops-sandbox.iam.gserviceaccount.com           False\n",
      "Service Account: vertex-pipelines already exists in project automlops-sandbox\n",
      "\u001b[0;32m Setting up IAM roles for Pipeline Job Runner Service Account in project automlops-sandbox \u001b[0m\n",
      "\u001b[0;32m Setting up Cloud Source Repository in project automlops-sandbox \u001b[0m\n",
      "Creating Cloud Source Repository: iris-k-means-repository in project automlops-sandbox\n",
      "Created [iris-k-means-repository].\n",
      "WARNING: You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.\n",
      "\u001b[0;32m Setting up Queueing Service in project automlops-sandbox \u001b[0m\n",
      "Creating Pub/Sub Topic: iris-k-means-queueing-svc in project automlops-sandbox\n",
      "Created topic [projects/automlops-sandbox/topics/iris-k-means-queueing-svc].\n",
      "\u001b[0;32m Deploying Cloud Functions: iris-k-means-job-submission-svc in project automlops-sandbox \u001b[0m\n",
      "Deploying function (may take a while - up to 2 minutes)...\n",
      "..\n",
      "For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/1793a967-68ce-49be-9bd1-f8ae1f1ddc2e?project=45373616427\n",
      "..............................................................done.\n",
      "availableMemoryMb: 512\n",
      "buildId: 1793a967-68ce-49be-9bd1-f8ae1f1ddc2e\n",
      "buildName: projects/45373616427/locations/us-central1/builds/1793a967-68ce-49be-9bd1-f8ae1f1ddc2e\n",
      "dockerRegistry: CONTAINER_REGISTRY\n",
      "entryPoint: process_request\n",
      "eventTrigger:\n",
      "  eventType: google.pubsub.topic.publish\n",
      "  failurePolicy: {}\n",
      "  resource: projects/automlops-sandbox/topics/iris-k-means-queueing-svc\n",
      "  service: pubsub.googleapis.com\n",
      "ingressSettings: ALLOW_ALL\n",
      "labels:\n",
      "  deployment-tool: cli-gcloud\n",
      "maxInstances: 3000\n",
      "name: projects/automlops-sandbox/locations/us-central1/functions/iris-k-means-job-submission-svc\n",
      "runtime: python39\n",
      "serviceAccountEmail: vertex-pipelines@automlops-sandbox.iam.gserviceaccount.com\n",
      "sourceUploadUrl: https://storage.googleapis.com/uploads-961973632599.us-central1.cloudfunctions.appspot.com/fed7b1ee-51c3-4e43-a03d-76d321fba63b.zip\n",
      "status: ACTIVE\n",
      "timeout: 540s\n",
      "updateTime: '2023-09-12T23:59:51.461Z'\n",
      "versionId: '1'\n",
      "\u001b[0;32m Setting up Cloud Build Trigger in project automlops-sandbox \u001b[0m\n",
      "Creating Cloudbuild Trigger on branch automlops in project automlops-sandbox for repo iris-k-means-repository\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/automlops-sandbox/locations/us-central1/triggers/8706c8c0-25ae-4a20-813d-8b1e9a5463b0].\n",
      "NAME                        CREATE_TIME                STATUS\n",
      "iris-k-means-build-trigger  2023-09-12T23:59:54+00:00\n",
      "\u001b[0;32m Setting up Cloud Scheduler Job in project automlops-sandbox \u001b[0m\n",
      "Creating Cloud Scheduler Job: iris-k-means-schedule in project automlops-sandbox\n",
      "name: projects/automlops-sandbox/locations/us-central1/jobs/iris-k-means-schedule\n",
      "pubsubTarget:\n",
      "  data: ewogICAgImRhdGFfcGF0aCI6ICJnczovL2F1dG9tbG9wcy1zYW5kYm94LWJ1Y2tldC9rbWVhbnMvMjAyMy0wOS0xMiAxOTo1NzozNS4wMDU3MjMvaXJpcy5jc3YiLAogICAgImNsdXN0ZXJfcGF0aCI6ICJnczovL2F1dG9tbG9wcy1zYW5kYm94LWJ1Y2tldC9rbWVhbnMvMjAyMy0wOS0xMiAxOTo1NzozNS4wMDU3MjMvaXJpc19jbHVzdGVycy5jc3YiLAogICAgImdzX3BpcGVsaW5lX3NwZWNfcGF0aCI6ICJnczovL2F1dG9tbG9wcy1zYW5kYm94LWlyaXMtay1tZWFucy1idWNrZXQvcGlwZWxpbmVfcm9vdC9pcmlzLWstbWVhbnMvcGlwZWxpbmVfam9iLmpzb24iCn0=\n",
      "  topicName: projects/automlops-sandbox/topics/iris-k-means-queueing-svc\n",
      "retryConfig:\n",
      "  maxBackoffDuration: 3600s\n",
      "  maxDoublings: 16\n",
      "  maxRetryDuration: 0s\n",
      "  minBackoffDuration: 5s\n",
      "schedule: 59 11 * * 0\n",
      "state: ENABLED\n",
      "timeZone: Etc/UTC\n",
      "userUpdateTime: '2023-09-12T23:59:56Z'\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.provision()            # hide_warnings is optional, defaults to True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40933839",
   "metadata": {},
   "source": [
    "`AutoMLOps.deploy(...)` builds and pushes component container, then triggers the pipeline job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "35666886",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized empty Git repository in /Users/srastatter/Documents/2023/MLOps-graduation/AutoMLOps-github/examples/training/.git/\n",
      "Switched to a new branch 'automlops'\n",
      "[automlops (root-commit) 007af6e] init\n",
      " 1 file changed, 149 insertions(+)\n",
      " create mode 100644 .gitignore\n",
      "remote: Waiting for private key checker: 1/1 objects left        \n",
      "To https://source.developers.google.com/p/automlops-sandbox/r/iris-k-means-repository\n",
      " * [new branch]      automlops -> automlops\n",
      "[automlops 44f89f9] Run AutoMLOps\n",
      " 23 files changed, 1031 insertions(+)\n",
      " create mode 100644 AutoMLOps/README.md\n",
      " create mode 100644 AutoMLOps/cloudbuild.yaml\n",
      " create mode 100644 AutoMLOps/components/component_base/Dockerfile\n",
      " create mode 100644 AutoMLOps/components/component_base/requirements.txt\n",
      " create mode 100644 AutoMLOps/components/component_base/src/create_dataset.py\n",
      " create mode 100644 AutoMLOps/components/component_base/src/fit_kmeans.py\n",
      " create mode 100644 AutoMLOps/components/create_dataset/component.yaml\n",
      " create mode 100644 AutoMLOps/components/fit_kmeans/component.yaml\n",
      " create mode 100644 AutoMLOps/configs/defaults.yaml\n",
      " create mode 100644 AutoMLOps/pipelines/pipeline.py\n",
      " create mode 100644 AutoMLOps/pipelines/pipeline_runner.py\n",
      " create mode 100644 AutoMLOps/pipelines/requirements.txt\n",
      " create mode 100644 AutoMLOps/pipelines/runtime_parameters/pipeline_parameter_values.json\n",
      " create mode 100755 AutoMLOps/provision/provision_resources.sh\n",
      " create mode 100755 AutoMLOps/scripts/build_components.sh\n",
      " create mode 100755 AutoMLOps/scripts/build_pipeline_spec.sh\n",
      " create mode 100644 AutoMLOps/scripts/pipeline_spec/.gitkeep\n",
      " create mode 100755 AutoMLOps/scripts/publish_to_topic.sh\n",
      " create mode 100755 AutoMLOps/scripts/run_all.sh\n",
      " create mode 100755 AutoMLOps/scripts/run_pipeline.sh\n",
      " create mode 100644 AutoMLOps/services/submission_service/Dockerfile\n",
      " create mode 100644 AutoMLOps/services/submission_service/main.py\n",
      " create mode 100644 AutoMLOps/services/submission_service/requirements.txt\n",
      "remote: Waiting for private key checker: 22/23 objects left        \n",
      "To https://source.developers.google.com/p/automlops-sandbox/r/iris-k-means-repository\n",
      "   007af6e..44f89f9  automlops -> automlops\n",
      "Pushing code to automlops branch, triggering build...\n",
      "Cloud Build job running at: https://console.cloud.google.com/cloud-build/builds;region=us-central1\n",
      "Please wait for this build job to complete.\n",
      "\n",
      "#################################################################\n",
      "#                                                               #\n",
      "#                       RESOURCES MANIFEST                      #\n",
      "#---------------------------------------------------------------#\n",
      "#     Generated resources can be found at the following urls    #\n",
      "#                                                               #\n",
      "#################################################################\n",
      "\n",
      "Google Cloud Storage Bucket: https://console.cloud.google.com/storage/automlops-sandbox-iris-k-means-bucket\n",
      "Artifact Registry: https://console.cloud.google.com/artifacts/docker/automlops-sandbox/us-central1/iris-k-means-artifact-registry\n",
      "Service Accounts: https://console.cloud.google.com/iam-admin/serviceaccounts?project=automlops-sandbox\n",
      "APIs: https://console.cloud.google.com/apis\n",
      "Cloud Source Repository: https://source.cloud.google.com/automlops-sandbox/iris-k-means-repository/+/automlops:\n",
      "Cloud Build Jobs: https://console.cloud.google.com/cloud-build/builds;region=us-central1\n",
      "Vertex AI Pipeline Runs: https://console.cloud.google.com/vertex-ai/pipelines/runs\n",
      "Cloud Build Trigger: https://console.cloud.google.com/cloud-build/triggers;region=us-central1\n",
      "Pipeline Job Submission Service (Cloud Functions): https://console.cloud.google.com/functions/details/us-central1/iris-k-means-job-submission-svc\n",
      "Pub/Sub Queueing Service Topic: https://console.cloud.google.com/cloudpubsub/topic/detail/iris-k-means-queueing-svc\n",
      "Pub/Sub Queueing Service Subscriptions: https://console.cloud.google.com/cloudpubsub/subscription/list\n",
      "Cloud Scheduler Job: https://console.cloud.google.com/cloudscheduler\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.deploy()                     # precheck is optional, defaults to True\n",
    "                                       # hide_warnings is optional, defaults to True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "483e8cac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "common-cpu.m102",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m102"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}