   ml_ops/sm-mlflow_pipelines/sm-mlflow_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "32cbb17e-1967-45c0-a564-385fe2136dc0",
   "metadata": {},
   "source": [
    "# SageMaker Pipelines with MLflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0908eda",
   "metadata": {},
   "source": [
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a10420-4496-4d45-ba93-c261c3ca2aa8",
   "metadata": {},
   "source": [
    "## Setup environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfb5edce-c55d-4cdf-afb0-ed20a31dcba4",
   "metadata": {},
   "source": [
    "Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e0aab5a-5d0c-48da-a3c2-1488a58d59d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker.workflow.execution_variables import ExecutionVariables\n",
    "from sagemaker.workflow.function_step import step\n",
    "from sagemaker.workflow.parameters import ParameterString\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.fail_step import FailStep"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d564fcd3-e7a3-4e0d-97a2-fcd9f4c9df6e",
   "metadata": {},
   "source": [
    "Declare some variables used later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0f4c80-563b-42c9-8a7c-f101d6eb0385",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.session.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"{default_bucket_prefix}/\"\n",
    "\n",
    "region = sagemaker_session.boto_region_name\n",
    "\n",
    "pipeline_name = \"breast-cancer-xgb\"\n",
    "instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.m5.xlarge\")\n",
    "\n",
    "# Mlflow (replace these values with your own)\n",
    "tracking_server_arn = \"your tracking server arn\"\n",
    "experiment_name = \"sm-pipelines-experiment\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2b42762-4725-41bd-92b5-019fa158b22d",
   "metadata": {
    "tags": []
   },
   "source": [
    "Write `requirements` and `config` files that'll be used by the steps in our SageMaker Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4aebe8c-c500-48c7-9204-34217b3cc424",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile config.yaml\n",
    "SchemaVersion: '1.0'\n",
    "SageMaker:\n",
    "  PythonSDK:\n",
    "    Modules:\n",
    "      RemoteFunction:\n",
    "        # role arn is not required if in SageMaker Notebook instance or SageMaker Studio\n",
    "        # Uncomment the following line and replace with the right execution role if in a local IDE\n",
    "        # RoleArn: <replace the role arn here>\n",
    "        InstanceType: ml.m5.xlarge\n",
    "        Dependencies: ./requirements.txt\n",
    "        IncludeLocalWorkDir: true\n",
    "        CustomFileFilter:\n",
    "          IgnoreNamePatterns: # files or directories to ignore\n",
    "          - \"*.ipynb\" # all notebook files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79a2d4d8-6ff5-4e87-afbb-48290f8eebeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set path to config file\n",
    "os.environ[\"SAGEMAKER_USER_CONFIG_OVERRIDE\"] = os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b8ea2db-85b2-4af8-9b6f-82ccbe104738",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile requirements.txt\n",
    "scikit-learn\n",
    "xgboost==1.7.6\n",
    "s3fs==0.4.2\n",
    "sagemaker>=2.199.0,<3\n",
    "pandas>=2.0.0\n",
    "gevent\n",
    "geventhttpclient\n",
    "shap\n",
    "matplotlib\n",
    "fsspec\n",
    "mlflow==2.13.2\n",
    "sagemaker-mlflow==0.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11a4daec-3a2d-4538-8e96-a10024755117",
   "metadata": {},
   "source": [
    "## Define the SageMaker Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "072dbf47-2fb2-4b9e-a64d-a22409d1b473",
   "metadata": {},
   "source": [
    "### Preprocessing Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da930770-7efc-4000-b104-8fc94c7133a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Location of our dataset\n",
    "input_path = f\"s3://sagemaker-example-files-prod-{region}/datasets/tabular/breast_cancer/wdbc.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "802af4a5-1813-4974-81a9-a22518b584a0",
   "metadata": {},
   "source": [
    "The breast cancer Wisconsin dataset contains column `id` which we do not use for training. The second column `diagnosis` is class label, and the label is represented using 'M' for Malignant class, and 'B' for Benign class. \n",
    "\n",
    "In the preprocessing step, we drop the column `id`, then split the dataset into three distinct sets: train, validation, and test set.\n",
    "\n",
    "Note that `keep_alive_period_in_seconds` parameter in @step decorator indicates how many seconds we want to keep the instance alive, waiting to be reused for the next pipeline step execution. Setting this parameter speeds up the pipeline execution because we reduce the launching of new instances to execute pipeline steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c54f225-f5c4-4766-a8dd-7e6c07554422",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "random_state = 2023\n",
    "label_column = \"diagnosis\"\n",
    "\n",
    "feature_names = [\n",
    "    \"id\",\n",
    "    \"diagnosis\",\n",
    "    \"radius_mean\",\n",
    "    \"texture_mean\",\n",
    "    \"perimeter_mean\",\n",
    "    \"area_mean\",\n",
    "    \"smoothness_mean\",\n",
    "    \"compactness_mean\",\n",
    "    \"concavity_mean\",\n",
    "    \"concave points_mean\",\n",
    "    \"symmetry_mean\",\n",
    "    \"fractal_dimension_mean\",\n",
    "    \"radius_se\",\n",
    "    \"texture_se\",\n",
    "    \"perimeter_se\",\n",
    "    \"area_se\",\n",
    "    \"smoothness_se\",\n",
    "    \"compactness_se\",\n",
    "    \"concavity_se\",\n",
    "    \"concave points_se\",\n",
    "    \"symmetry_se\",\n",
    "    \"fractal_dimension_se\",\n",
    "    \"radius_worst\",\n",
    "    \"texture_worst\",\n",
    "    \"perimeter_worst\",\n",
    "    \"area_worst\",\n",
    "    \"smoothness_worst\",\n",
    "    \"compactness_worst\",\n",
    "    \"concavity_worst\",\n",
    "    \"concave points_worst\",\n",
    "    \"symmetry_worst\",\n",
    "    \"fractal_dimension_worst\",\n",
    "]\n",
    "\n",
    "\n",
    "@step(\n",
    "    name=\"DataPreprocessing\",\n",
    "    instance_type=instance_type,\n",
    ")\n",
    "def preprocess(\n",
    "    raw_data_s3_path: str,\n",
    "    output_prefix: str,\n",
    "    experiment_name: str,\n",
    "    run_name: str,\n",
    "    test_size: float = 0.2,\n",
    ") -> tuple:\n",
    "    import mlflow\n",
    "    import pandas as pd\n",
    "    from sklearn.model_selection import train_test_split\n",
    "\n",
    "    mlflow.set_tracking_uri(tracking_server_arn)\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "    with mlflow.start_run(run_name=run_name) as run:\n",
    "        run_id = run.info.run_id\n",
    "        with mlflow.start_run(run_name=\"DataPreprocessing\", nested=True):\n",
    "            df = pd.read_csv(raw_data_s3_path, header=None, names=feature_names)\n",
    "            df.drop(columns=\"id\", inplace=True)\n",
    "            mlflow.log_input(\n",
    "                mlflow.data.from_pandas(df, raw_data_s3_path, targets=label_column),\n",
    "                context=\"DataPreprocessing\",\n",
    "            )\n",
    "\n",
    "            train_df, test_df = train_test_split(df, test_size=0.2, stratify=df[label_column])\n",
    "            validation_df, test_df = train_test_split(\n",
    "                test_df, test_size=0.5, stratify=test_df[label_column]\n",
    "            )\n",
    "            train_df.reset_index(inplace=True, drop=True)\n",
    "            validation_df.reset_index(inplace=True, drop=True)\n",
    "            test_df.reset_index(inplace=True, drop=True)\n",
    "\n",
    "            train_s3_path = f\"s3://{bucket}/{output_prefix}/train.csv\"\n",
    "            val_s3_path = f\"s3://{bucket}/{output_prefix}/val.csv\"\n",
    "            test_s3_path = f\"s3://{bucket}/{output_prefix}/test.csv\"\n",
    "\n",
    "            train_df.to_csv(train_s3_path, index=False)\n",
    "            validation_df.to_csv(val_s3_path, index=False)\n",
    "            test_df.to_csv(test_s3_path, index=False)\n",
    "\n",
    "    return train_s3_path, val_s3_path, test_s3_path, experiment_name, run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a99152-3495-45fc-abc9-d929bb0066d5",
   "metadata": {},
   "source": [
    "### Training Step"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da081365-db2d-492f-b1e0-ab9a4ff8a99f",
   "metadata": {},
   "source": [
    "We train a XGBoost model in this training step, using @step-decorated function with the S3 path of training and validation set, along with XGBoost hyperparameters. The S3 paths for both training and validation set is coming from the output of the previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb98b312-208a-4327-add7-f80184c4d392",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "use_gpu = False\n",
    "param = dict(\n",
    "    objective=\"binary:logistic\",\n",
    "    max_depth=5,\n",
    "    eta=0.2,\n",
    "    gamma=4,\n",
    "    min_child_weight=6,\n",
    "    subsample=0.7,\n",
    "    tree_method=\"gpu_hist\" if use_gpu else \"hist\",  # Use GPU accelerated algorithm\n",
    ")\n",
    "num_round = 50\n",
    "\n",
    "\n",
    "@step(\n",
    "    name=\"ModelTraining\",\n",
    "    instance_type=instance_type,\n",
    ")\n",
    "def train(\n",
    "    train_s3_path: str,\n",
    "    validation_s3_path: str,\n",
    "    experiment_name: str,\n",
    "    run_id: str,\n",
    "    param: dict = param,\n",
    "    num_round: int = num_round,\n",
    "):\n",
    "    import mlflow\n",
    "    import pandas as pd\n",
    "    from xgboost import XGBClassifier\n",
    "\n",
    "    mlflow.set_tracking_uri(tracking_server_arn)\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "\n",
    "    with mlflow.start_run(run_id=run_id):\n",
    "        with mlflow.start_run(run_name=\"ModelTraining\", nested=True) as training_run:\n",
    "            training_run_id = training_run.info.run_id\n",
    "            mlflow.xgboost.autolog(\n",
    "                log_input_examples=True,\n",
    "                log_model_signatures=True,\n",
    "                log_models=True,\n",
    "                log_datasets=True,\n",
    "                model_format=\"xgb\",\n",
    "            )\n",
    "\n",
    "            # read data files from S3\n",
    "            train_df = pd.read_csv(train_s3_path)\n",
    "            validation_df = pd.read_csv(validation_s3_path)\n",
    "\n",
    "            # create dataframe and label series\n",
    "            y_train = (train_df.pop(label_column) == \"M\").astype(\"int\")\n",
    "            y_validation = (validation_df.pop(label_column) == \"M\").astype(\"int\")\n",
    "\n",
    "            xgb = XGBClassifier(n_estimators=num_round, **param)\n",
    "            xgb.fit(\n",
    "                train_df,\n",
    "                y_train,\n",
    "                eval_set=[(validation_df, y_validation)],\n",
    "                early_stopping_rounds=5,\n",
    "            )\n",
    "\n",
    "        # return xgb\n",
    "        return experiment_name, run_id, training_run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add79f19-fa80-45e6-9c5e-3d454954fa63",
   "metadata": {},
   "source": [
    "### Evaluation Step"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe4709c-106a-4a63-ae49-09d24ab35262",
   "metadata": {},
   "source": [
    "In this step, we create a @step-decorated function to evaluate the trained XGBoost model on the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e0cae65-6859-433c-9b17-a3c5d1a881a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@step(\n",
    "    name=\"ModelEvaluation\",\n",
    "    instance_type=instance_type,\n",
    ")\n",
    "def evaluate(\n",
    "    test_s3_path: str,\n",
    "    experiment_name: str,\n",
    "    run_id: str,\n",
    "    training_run_id: str,\n",
    ") -> dict:\n",
    "    import mlflow\n",
    "    import pandas as pd\n",
    "\n",
    "    mlflow.set_tracking_uri(tracking_server_arn)\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "\n",
    "    with mlflow.start_run(run_id=run_id):\n",
    "        with mlflow.start_run(run_name=\"ModelEvaluation\", nested=True):\n",
    "            test_df = pd.read_csv(test_s3_path)\n",
    "            test_df[label_column] = (test_df[label_column] == \"M\").astype(\"int\")\n",
    "            model = mlflow.pyfunc.load_model(f\"runs:/{training_run_id}/model\")\n",
    "\n",
    "            results = mlflow.evaluate(\n",
    "                model=model,\n",
    "                data=test_df,\n",
    "                targets=label_column,\n",
    "                model_type=\"classifier\",\n",
    "                evaluators=[\"default\"],\n",
    "            )\n",
    "            return {\"f1_score\": results.metrics[\"f1_score\"]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47f9415-8edf-49a5-906b-6d112c24d0b3",
   "metadata": {},
   "source": [
    "### Model Registration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac789be1-e1d9-4102-bbf4-c3284e6dc062",
   "metadata": {},
   "source": [
    "In this step, we create a @step-decorated function to register our XGBoost model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9af124b1-1cae-49da-9854-8d1342af844f",
   "metadata": {},
   "outputs": [],
   "source": [
    "@step(\n",
    "    name=\"ModelRegistration\",\n",
    "    instance_type=instance_type,\n",
    ")\n",
    "def register(\n",
    "    pipeline_name: str,\n",
    "    experiment_name: str,\n",
    "    run_id: str,\n",
    "    training_run_id: str,\n",
    "):\n",
    "    import mlflow\n",
    "\n",
    "    mlflow.set_tracking_uri(tracking_server_arn)\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "\n",
    "    with mlflow.start_run(run_id=run_id):\n",
    "        with mlflow.start_run(run_name=\"ModelRegistration\", nested=True):\n",
    "            mlflow.register_model(f\"runs:/{training_run_id}/model\", pipeline_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c736d1-2aa3-4bb5-a235-be56fe07e875",
   "metadata": {},
   "source": [
    "## Creating the SageMaker Pipeline\n",
    "\n",
    "We connect all defined pipeline `@step` functions into a multi-step pipeline. Then, we submit and execute the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5339f24f-15ec-4a6c-8f98-d8c191fabf4c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preprocessing_step = preprocess(\n",
    "    raw_data_s3_path=input_path,\n",
    "    output_prefix=f\"{default_bucket_prefix_path}{pipeline_name}/dataset\",\n",
    "    experiment_name=experiment_name,\n",
    "    run_name=ExecutionVariables.PIPELINE_EXECUTION_ID,\n",
    ")\n",
    "\n",
    "training_step = train(\n",
    "    train_s3_path=preprocessing_step[0],\n",
    "    validation_s3_path=preprocessing_step[1],\n",
    "    experiment_name=preprocessing_step[3],\n",
    "    run_id=preprocessing_step[4],\n",
    ")\n",
    "\n",
    "conditional_register_step = ConditionStep(\n",
    "    name=\"ConditionalRegister\",\n",
    "    conditions=[\n",
    "        ConditionGreaterThanOrEqualTo(\n",
    "            left=evaluate(\n",
    "                test_s3_path=preprocessing_step[2],\n",
    "                experiment_name=preprocessing_step[3],\n",
    "                run_id=preprocessing_step[4],\n",
    "                training_run_id=training_step[2],\n",
    "            )[\"f1_score\"],\n",
    "            right=0.8,\n",
    "        )\n",
    "    ],\n",
    "    if_steps=[\n",
    "        register(\n",
    "            pipeline_name=pipeline_name,\n",
    "            experiment_name=preprocessing_step[3],\n",
    "            run_id=preprocessing_step[4],\n",
    "            training_run_id=training_step[2],\n",
    "        )\n",
    "    ],\n",
    "    else_steps=[FailStep(name=\"Fail\", error_message=\"Model performance is not good enough\")],\n",
    ")\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    parameters=[\n",
    "        instance_type,\n",
    "    ],\n",
    "    steps=[preprocessing_step, training_step, conditional_register_step],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7efafac7-5865-4ffa-b410-4d1ad7e3a576",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline.upsert(role_arn=role)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3668a3ed-207a-4c19-bbe2-61ef778605a0",
   "metadata": {},
   "source": [
    "## Execute the SageMaker Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4b7a103-783e-409b-8bc3-583e587610bd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution = pipeline.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3350472d-ffd6-41ed-8a1d-ded0318feeae",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef26ee0-c1d2-4a43-9884-cf32032f5b2b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "894beeea-37fb-49ec-9627-713c9d9f4040",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a17b4de5",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ml_ops|sm-mlflow_pipelines|sm-mlflow_pipelines.ipynb)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}