 archived/SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Clarify Model Bias Monitor for Batch Transform"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Runtime\n",
    "\n",
    "This notebook takes approximately 60 minutes to run."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "* [Introduction](#Introduction)\n",
    "* [General Setup](#General-Setup)\n",
    "    * [Imports](#Imports)\n",
    "    * [Handful of configuration](#Handful-of-configuration)\n",
    "    * [Data files](#Data-files)\n",
    "    * [SageMaker model](#SageMaker-model)\n",
    "* [Batch Transform Job](#Batch-Transform-Job)\n",
    "    * [Captured data](#Captured-data)\n",
    "    * [Transform output](#Transform-output)\n",
    "* [Ground Truth Data](#Ground-Truth-Data)\n",
    "* [Model Bias Monitor](#Model-Bias-Monitor)\n",
    "    * [Baselining job](#Baselining-job)\n",
    "        * [Configurations](#Configurations)\n",
    "        * [Kick off baselining job](#Kick-off-baselining-job)\n",
    "    * [Monitoring Schedule](#Monitoring-Schedule)\n",
    "        * [Wait for the first execution](#Wait-for-the-first-execution)\n",
    "        * [Wait for the execution to finish](#Wait-for-the-execution-to-finish)\n",
    "        * [Merged data](#Merged-data)\n",
    "        * [Inspect execution results](#Inspect-execution-results)\n",
    "* [Cleanup](#Cleanup)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Introduction\n",
    "\n",
    "[Amazon SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html) continuously monitors the quality of Amazon SageMaker machine learning models in production. It enables developers to set alerts for when there are deviations in the model quality. Early and pro-active detection of these deviations enables corrective actions, such as retraining models, auditing upstream systems, or fixing data quality issues without having to monitor models manually or build additional tooling. \n",
    "\n",
    "[Amazon SageMaker Clarify Model Bias Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-bias-drift.html) is a model monitor that helps data scientists and ML engineers monitor predictions for bias on a regular basis. Bias can be introduced or exacerbated in deployed ML models when the training data differs from the data that the model sees during deployment (that is, the live data). These kinds of changes in the live data distribution might be temporary (for example, due to some short-lived, real-world events) or permanent. In either case, it might be important to detect these changes. For example, the outputs of a model for predicting home prices can become biased if the mortgage rates used to train the model differ from current, real-world mortgage rates. With bias drift detection capabilities in model monitor, when SageMaker detects bias beyond a certain threshold, it automatically generates metrics that you can view in SageMaker Studio and through Amazon CloudWatch alerts. \n",
    "\n",
    "This notebook demonstrates the process for setting up a model monitor for continuous monitoring of bias drift of the data and model used by a regularly running [SageMaker Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) job. The model input and output are in CSV format.\n",
    "\n",
    "In general, you can use the model bias monitor for batch transform in this way,\n",
    "\n",
    "1. Schedule a model bias monitor to monitor a data capture S3 location and a ground truth S3 location\n",
    "1. Regularly run transform jobs with data capture enabled, the jobs save captured data to the data capture S3 URI\n",
    "1. Regularly label the captured data, and then upload the ground truth labels to the ground truth S3 URI\n",
    "\n",
    "The monitor executes processing jobs regularly to merge the captured data and ground truth data, do bias analysis for the merged data, and then generate analysis reports and publish metrics to CloudWatch."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## General Setup"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook uses the [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk). The following cell upgrades the SDK and its dependencies. Then you may need to restart the kernel and rerun the notebook to pick up the up-to-date APIs, if the notebook is executed in the SageMaker Studio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -U sagemaker\n",
    "!pip install -U boto3\n",
    "!pip install -U botocore"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports\n",
    "\n",
    "The following cell imports the APIs to be used by the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import pandas as pd\n",
    "import datetime\n",
    "import io\n",
    "import json\n",
    "import os\n",
    "import random\n",
    "import time\n",
    "import pprint"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handful of configuration\n",
    "\n",
    "To begin, ensure that these prerequisites have been completed.\n",
    "\n",
    "* Specify an AWS Region to host the model.\n",
    "* Specify an IAM role to execute jobs.\n",
    "* Define the S3 URIs that stores the model file, input data and output data. For demonstration purposes, this notebook uses the same bucket for them. In reality, they could be separated with different security policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "region = sagemaker_session.boto_region_name\n",
    "print(f\"AWS region: {region}\")\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "print(f\"RoleArn: {role}\")\n",
    "\n",
    "# A different bucket can be used, but make sure the role for this notebook has\n",
    "# the s3:PutObject permissions. This is the bucket into which the data is captured\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "print(f\"Demo Bucket: {bucket}\")\n",
    "prefix = sagemaker.utils.unique_name_from_base(\"sagemaker/DEMO-ClarifyModelMonitor\")\n",
    "print(f\"Demo Prefix: {prefix}\")\n",
    "s3_key = f\"s3://{bucket}/{prefix}\"\n",
    "print(f\"Demo S3 key: {s3_key}\")\n",
    "\n",
    "data_capture_s3_uri = f\"{s3_key}/data-capture\"\n",
    "ground_truth_s3_uri = f\"{s3_key}/ground-truth\"\n",
    "transform_output_s3_uri = f\"{s3_key}/transform-output\"\n",
    "baselining_output_s3_uri = f\"{s3_key}/baselining-output\"\n",
    "monitor_output_s3_uri = f\"{s3_key}/monitor-output\"\n",
    "\n",
    "print(f\"The transform job will save the results to: {transform_output_s3_uri}\")\n",
    "print(f\"The transform job will save the captured data to: {data_capture_s3_uri}\")\n",
    "print(f\"You should upload the ground truth data to: {ground_truth_s3_uri}\")\n",
    "print(f\"The baselining job will save the analysis results to: {baselining_output_s3_uri}\")\n",
    "print(f\"The monitor will save the analysis results to: {monitor_output_s3_uri}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Data files\n",
    "\n",
    "This example includes two dataset files, both in CSV format.\n",
    "\n",
    "* The train dataset has header row, and it has a target column followed by the feature columns.\n",
    "* The test dataset is not headers, and it only has feature columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_dataset_path = \"test_data/validation-dataset-with-header.csv\"\n",
    "test_dataset_path = \"test_data/test-dataset-input-cols.csv\"\n",
    "dataset_type = \"text/csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(train_dataset_path)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Read headers\n",
    "all_headers = list(df.columns)\n",
    "label_header = all_headers[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To verify that the execution role for this notebook has the necessary permissions to proceed, put a simple test object into the S3 bucket specified above. If this command fails, update the role to have `s3:PutObject` permission on the bucket and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sagemaker.s3.S3Uploader.upload_string_as_file_body(\n",
    "    body=\"hello\",\n",
    "    desired_s3_uri=f\"{s3_key}/upload-test-file.txt\",\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "print(\"Success! We are all set to proceed with uploading to S3.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then upload the data files to S3 so that they can be used by SageMaker jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data_s3_uri = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=train_dataset_path,\n",
    "    desired_s3_uri=s3_key,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "print(f\"Train data is uploaded to: {train_data_s3_uri}\")\n",
    "test_data_s3_uri = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=test_dataset_path,\n",
    "    desired_s3_uri=s3_key,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "print(f\"Test data is uploaded to: {test_data_s3_uri}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SageMaker model\n",
    "\n",
    "This example includes a pre-built [SageMaker XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) model file trained by [XGBoost Churn Prediction Notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb). The following cell uploads the file to S3 and then creates a SageMaker model using it. The model support CSV data format, the input are customer attributes, and the output is the probability of customer churn (a float number between zero and one)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_file = \"model/xgb-churn-prediction-model.tar.gz\"\n",
    "model_url = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=model_file,\n",
    "    desired_s3_uri=s3_key,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "print(f\"Model file has been uploaded to {model_url}\")\n",
    "\n",
    "model_name = sagemaker.utils.unique_name_from_base(\"DEMO-xgb-churn-pred-model-monitor\")\n",
    "print(f\"SageMaker model name: {model_name}\")\n",
    "\n",
    "image_uri = sagemaker.image_uris.retrieve(\"xgboost\", region, \"0.90-1\")\n",
    "print(f\"SageMaker XGBoost image: {image_uri}\")\n",
    "\n",
    "model = sagemaker.model.Model(image_uri=image_uri, model_data=model_url, role=role)\n",
    "container_def = model.prepare_container_def()\n",
    "sagemaker_session.create_model(model_name, role, container_def)\n",
    "print(\"SageMaker model created\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Batch Transform Job\n",
    "\n",
    "For continuous monitoring, batch transform jobs should be executed regularly with the latest data. But for demonstration purpose, the following cell only executes the job once before the monitor is scheduled, so that the first monitoring execution has captured data to process. \n",
    "\n",
    "See [Transformer](https://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html#sagemaker.transformer.Transformer.transform) for the API reference. Highlights,\n",
    "\n",
    "* `destination_s3_uri` is used to specify the data capture S3 URI which is a key connection between the job and the monitor.\n",
    "* `join_source` must be set to \"Input\" for the transform output to include predictions (model output) as well as features (model input), because model bias monitor requires both.\n",
    "* `generate_inference_id` must be set to True for the transform output to include a unique ID for each record. Model bias monitor requires both predicted labels and ground truth labels, so it needs the ID to join the captured data and the ground truth data.\n",
    "\n",
    "**NOTE**: The following cell takes about 5 minutes to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transfomer = model.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    accept=dataset_type,  # The transform output data format\n",
    "    assemble_with=\"Line\",  # CSV records are terminated by new lines\n",
    "    output_path=transform_output_s3_uri,\n",
    ")\n",
    "\n",
    "transfomer.transform(\n",
    "    data=test_data_s3_uri,\n",
    "    content_type=dataset_type,  # The transform input format\n",
    "    split_type=\"Line\",  # CSV records are terminated by new lines\n",
    "    join_source=\"Input\",  # Include model input (features) in transform output\n",
    "    batch_data_capture_config=sagemaker.inputs.BatchDataCaptureConfig(\n",
    "        destination_s3_uri=data_capture_s3_uri,\n",
    "        generate_inference_id=True,  # Inference ID is mandatory to join the captured data and the ground truth data\n",
    "    ),\n",
    "    wait=True,  # In real world you don't have to wait, but for demo purpose we wait for the output\n",
    "    logs=False,  # You can change it to True to view job logs inline\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Captured data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the transform job completed, an \"output\" folders is created under `data_capture_s3_uri`, to includes the captured data files of transform output. Note that, batch transform data capture is unlike endpoint data capture, it does not capture the data for real as it will create tremendous amount of duplications. Instead, it generates [manifest](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_S3DataSource.html#sagemaker-Type-S3DataSource-S3Uri) files which refer to the transform output S3 location."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now list the captured data files stored in Amazon S3. There should be different files from different time periods organized based on the hour in which the batch transformation occurred. The format of the Amazon S3 path is:\n",
    "\n",
    "`s3://{data_capture_s3_uri}/output/yyyy/mm/dd/hh/filename.jsonl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_capture_output = f\"{data_capture_s3_uri}/output\"\n",
    "captured_data_files = sorted(\n",
    "    sagemaker.s3.S3Downloader.list(\n",
    "        s3_uri=data_capture_output,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    ")\n",
    "print(\"Found capture data files:\")\n",
    "print(\"\\n \".join(captured_data_files[-5:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_capture_output_dict = json.loads(\n",
    "    sagemaker.s3.S3Downloader.read_file(\n",
    "        s3_uri=captured_data_files[-1],\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    ")\n",
    "print(json.dumps(data_capture_output_dict, indent=4))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform output\n",
    "\n",
    "The captured data file refers to the transform output `.out` file. The cell below shows the first few records of the file.\n",
    "\n",
    "* The first columns are the feature columns (the model input), because the `join_source` parameter is set to \"Input\".\n",
    "* Then there is the prediction column (the model output).\n",
    "* The second last element is the inference ID, and the last is the inference time (the start time of the transform job). They are available because the `generate_inference_id` parameter is set to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transform_output = os.path.join(data_capture_output_dict[0][\"prefix\"], data_capture_output_dict[1])\n",
    "transform_output_content = sagemaker.s3.S3Downloader.read_file(\n",
    "    s3_uri=transform_output,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "transform_output_df = pd.read_csv(io.StringIO(transform_output_content), header=None)\n",
    "transform_output_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Ground Truth Data\n",
    "\n",
    "Besides captured data, bias drift monitoring execution also requires ground truth data. In real use cases, you should regularly label the captured data, then upload the ground truth data (labels) to designated S3 location. For demonstration purpose, this example notebook generates fake ground truth data following [this schema](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-merge.html), and then uploads it to `ground_truth_s3_uri` which is another key input to the monitor. The bias drift monitoring execution will first merge the captured data and the ground truth data, and then do bias analysis for the merged data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def ground_truth_with_id(inference_id):\n",
    "    random.seed(inference_id)  # to get consistent results\n",
    "    # format required by the merge job and bias monitoring job\n",
    "    return {\n",
    "        \"groundTruthData\": {\n",
    "            \"data\": \"1\"\n",
    "            if random.random() < 0.7\n",
    "            else \"0\",  # randomly generate positive labels 70% of the time\n",
    "            \"encoding\": \"CSV\",\n",
    "        },\n",
    "        \"eventMetadata\": {\n",
    "            \"eventId\": str(\n",
    "                inference_id\n",
    "            ),  # the id is used to join the captured data and the ground truth data\n",
    "        },\n",
    "        \"eventVersion\": \"0\",\n",
    "    }\n",
    "\n",
    "\n",
    "def upload_ground_truth(upload_time, upload_path, inference_ids):\n",
    "    records = [ground_truth_with_id(inference_id) for inference_id in inference_ids]\n",
    "    fake_records = [json.dumps(r) for r in records]\n",
    "    data_to_upload = \"\\n\".join(fake_records)\n",
    "    target_s3_uri = f\"{upload_path}/{upload_time:%Y/%m/%d/%H/%M%S}.jsonl\"\n",
    "    print(f\"Uploading {len(fake_records)} records to\", target_s3_uri)\n",
    "    sagemaker.s3.S3Uploader.upload_string_as_file_body(\n",
    "        body=data_to_upload,\n",
    "        desired_s3_uri=target_s3_uri,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "now = datetime.datetime.utcnow()\n",
    "inference_ids = list(transform_output_df.iloc[:, -2])  # get inference ID from the captured data\n",
    "# Generate data for the last hour, in case the first monitoring execution is in this hour\n",
    "upload_ground_truth(\n",
    "    upload_time=now - datetime.timedelta(hours=1),\n",
    "    upload_path=ground_truth_s3_uri,\n",
    "    inference_ids=inference_ids,\n",
    ")\n",
    "# Generate data for this hour, in case the first monitoring execution will be in the next hour\n",
    "upload_ground_truth(\n",
    "    upload_time=now,\n",
    "    upload_path=ground_truth_s3_uri,\n",
    "    inference_ids=inference_ids,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Bias Monitor\n",
    "\n",
    "Similar to the other monitoring types, the standard procedure of creating a [bias drift monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-bias-drift.html) is first run a baselining job, and then schedule the monitor.\n",
    "\n",
    "A bias drift monitoring execution starts a merge job that joins the captured data and ground truth data together using the inference ID. Then a SageMaker Clarify bias analysis job is started to compute all the [pre-training bias metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-data-bias.html) and [post-training bias metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-post-training-bias.html). on the merged data. The max execution time is divided equally between two jobs, the notebook is scheduling an hourly model bias monitor, so the `max_runtime_in_seconds` parameter should not exceed 1800 seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_monitor = sagemaker.model_monitor.ModelBiasMonitor(\n",
    "    role=role,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    max_runtime_in_seconds=1800,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baselining job\n",
    "\n",
    "A baselining job runs predictions on training dataset and suggests constraints. The `suggest_baseline()` method of `ModelBiasMonitor` starts a SageMaker Clarify processing job to generate the constraints.\n",
    "\n",
    "The step is not mandatory, but providing constraints file to the monitor can enable violations file generation."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Configurations\n",
    "\n",
    "Information about the input data need to be provided to the processor."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataConfig` stores information about the dataset to be analyzed. For example, the dataset file, its format (like CSV), headers and label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_config = sagemaker.clarify.DataConfig(\n",
    "    s3_data_input_path=train_data_s3_uri,\n",
    "    s3_output_path=baselining_output_s3_uri,\n",
    "    label=label_header,\n",
    "    headers=all_headers,\n",
    "    dataset_type=dataset_type,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ModelConfig` is configuration related to model to be used for inferencing. In order to compute post-training bias metrics, the computation needs to get inferences for the SageMaker model. To accomplish this, the processing job will use the model to create an ephemeral endpoint (also known as \"shadow endpoint\"). The processing job will delete the shadow endpoint after the computations are completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_config = sagemaker.clarify.ModelConfig(\n",
    "    model_name=model_name,  # The name of the SageMaker model\n",
    "    instance_type=\"ml.m5.xlarge\",  # The instance type of the shadow endpoint\n",
    "    instance_count=1,  # The instance count of the shadow endpoint\n",
    "    content_type=dataset_type,  # The data format of the model input\n",
    "    accept_type=dataset_type,  # The data format of the model output\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ModelPredictedLabelConfig` specifies how to extract predicted label from the model output. The example model returns a single probability value between `0` and `1`. So,\n",
    "\n",
    "* The `probability` parameter is set to zero, which is the index of the probability value in the CSV model output. \n",
    "* The `probability_threshold` parameter is used by post-training analysis to convert the probability/score to binary predicted label (`0` or `1`). The default value is 0.5. Here choose an arbitrary 0.8 cutoff, i.e. a probability value > 0.8 means customer will churn (predicted label `1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_predicted_label_config = sagemaker.clarify.ModelPredictedLabelConfig(\n",
    "    probability=0,  # The zero-based index of the probability (score) in model output\n",
    "    probability_threshold=0.8,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BiasConfig` is the configuration of the sensitive groups in the dataset. Typically, bias is measured by computing a metric and comparing it across groups. The group of interest is specified using the `facet` configuration. With the following facet, the baselining job will check if the model favors new customers (accounts created not far ago)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bias_config = sagemaker.clarify.BiasConfig(\n",
    "    label_values_or_threshold=[1],\n",
    "    facet_name=\"Account Length\",\n",
    "    facet_values_or_threshold=[100],\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Kick off baselining job\n",
    "\n",
    "Call the `suggest_baseline()` method to start the baselining job. The job computes all [pre-training bias metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-data-bias.html) and [post-training bias metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-post-training-bias.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_monitor.suggest_baseline(\n",
    "    bias_config=bias_config,\n",
    "    data_config=data_config,\n",
    "    model_config=model_config,\n",
    "    model_predicted_label_config=model_predicted_label_config,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: The following cell waits until the baselining job is completed (in about 10 minutes). It then inspects the suggested constraints. This step can be skipped, because the monitor to be scheduled will automatically pick up baselining job name and wait for it before monitoring execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_monitor.latest_baselining_job.wait(logs=False)\n",
    "print()\n",
    "model_bias_constraints = model_bias_monitor.suggested_constraints()\n",
    "print(f\"Suggested constraints: {model_bias_constraints.file_s3_uri}\")\n",
    "print(\n",
    "    sagemaker.s3.S3Downloader.read_file(\n",
    "        s3_uri=model_bias_constraints.file_s3_uri,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring Schedule\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With above constraints collected, now call `create_monitoring_schedule()` method to schedule an hourly model bias monitor."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a baselining job has been submitted, then the monitor object will automatically pick up the analysis configuration from the baselining job. But if the baselining step is skipped, or if the capture dataset has different nature than the training dataset, then analysis configuration has to be provided.\n",
    "\n",
    "`BiasAnalysisConfig` is a subset of the configuration of the baselining job, many options are not needed because,\n",
    "\n",
    "* Model bias monitor will merge the captured data and the ground truth data, and then use the merged data as the input dataset.\n",
    "* Capture data already includes predictions, so there is no need to create shadow endpoint.\n",
    "* Attributes like probability threshold are provided as part of `BatchTransformInput`.\n",
    "\n",
    "Highlights,\n",
    "\n",
    "* `data_capture_s3_uri` is the location of data captured by the batch transform job\n",
    "* `ground_truth_s3_uri` is the location of ground truth data\n",
    "* `probability_attribute` stores the index of the probability value in model output. (Similar to the `probability` parameter of `ModelPredictedLabelConfig`.)\n",
    "* `probability_threshold_attribute` is the same as the `probability_threshold` parameter of `ModelPredictedLabelConfig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.hourly()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_analysis_config = None\n",
    "if not model_bias_monitor.latest_baselining_job:\n",
    "    model_bias_analysis_config = sagemaker.clarify.BiasAnalysisConfig(\n",
    "        bias_config,\n",
    "        headers=all_headers,\n",
    "        label=label_header,\n",
    "    )\n",
    "model_bias_monitor.create_monitoring_schedule(\n",
    "    analysis_config=model_bias_analysis_config,\n",
    "    batch_transform_input=sagemaker.model_monitor.BatchTransformInput(\n",
    "        data_captured_destination_s3_uri=data_capture_s3_uri,\n",
    "        destination=\"/opt/ml/processing/transform\",\n",
    "        dataset_format=sagemaker.model_monitor.MonitoringDatasetFormat.csv(header=False),\n",
    "        probability_attribute=\"0\",\n",
    "        probability_threshold_attribute=0.8,\n",
    "        # look back 6 hour for transform job output.\n",
    "        start_time_offset=\"-PT6H\",\n",
    "        end_time_offset=\"-PT0H\",\n",
    "    ),\n",
    "    ground_truth_input=ground_truth_s3_uri,\n",
    "    output_s3_uri=monitor_output_s3_uri,\n",
    "    schedule_cron_expression=schedule_expression,\n",
    ")\n",
    "print(f\"Model bias monitoring schedule: {model_bias_monitor.monitoring_schedule_name}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for the first execution\n",
    "\n",
    "The schedule starts jobs at the previously specified intervals. Code below waits until time crosses the hour boundary (in UTC) to see executions kick off.\n",
    "\n",
    "Note: Even for an hourly schedule, Amazon SageMaker has a buffer period of 20 minutes to schedule executions. The execution might start in anywhere from zero to ~20 minutes from the hour boundary. This is expected and done for load balancing in the backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def wait_for_execution_to_start(model_monitor):\n",
    "    print(\n",
    "        \"An hourly schedule was created above and it will kick off executions ON the hour (plus 0 - 20 min buffer).\"\n",
    "    )\n",
    "\n",
    "    print(\"Waiting for the first execution to happen\", end=\"\")\n",
    "    schedule_desc = model_monitor.describe_schedule()\n",
    "    while \"LastMonitoringExecutionSummary\" not in schedule_desc:\n",
    "        schedule_desc = model_monitor.describe_schedule()\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        time.sleep(60)\n",
    "    print()\n",
    "    print(\"Done! Execution has been created\")\n",
    "\n",
    "    print(\"Now waiting for execution to start\", end=\"\")\n",
    "    while schedule_desc[\"LastMonitoringExecutionSummary\"][\"MonitoringExecutionStatus\"] in \"Pending\":\n",
    "        schedule_desc = model_monitor.describe_schedule()\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        time.sleep(10)\n",
    "\n",
    "    print()\n",
    "    print(\"Done! Execution has started\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: The following cell waits until the first monitoring execution is started. As explained above, the wait could take more than 60 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "wait_for_execution_to_start(model_bias_monitor)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In real world, a monitoring schedule is supposed to be active all the time. But in this example, it can be stopped to avoid incurring extra charges. A stopped schedule will not trigger further executions, but the ongoing execution will continue. And if needed, the schedule can be restarted by `start_monitoring_schedule()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_monitor.stop_monitoring_schedule()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for the execution to finish\n",
    "\n",
    "In the previous cell, the first execution has started. This section waits for the execution to finish so that its analysis results are available. Here are the possible terminal states and what each of them mean:\n",
    "\n",
    "* `Completed` - This means the monitoring execution completed, and no issues were found in the violations report.\n",
    "* `CompletedWithViolations` - This means the execution completed, but constraint violations were detected.\n",
    "* `Failed` - The monitoring execution failed, maybe due to client error (perhaps incorrect role permissions) or infrastructure issues. Further examination of `FailureReason` and `ExitMessage` is necessary to identify what exactly happened.\n",
    "* `Stopped` - job exceeded max runtime or was manually stopped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Waits for the schedule to have last execution in a terminal status.\n",
    "def wait_for_execution_to_finish(model_monitor):\n",
    "    schedule_desc = model_monitor.describe_schedule()\n",
    "    execution_summary = schedule_desc.get(\"LastMonitoringExecutionSummary\")\n",
    "    if execution_summary is not None:\n",
    "        print(\"Waiting for execution to finish\", end=\"\")\n",
    "        while execution_summary[\"MonitoringExecutionStatus\"] not in [\n",
    "            \"Completed\",\n",
    "            \"CompletedWithViolations\",\n",
    "            \"Failed\",\n",
    "            \"Stopped\",\n",
    "        ]:\n",
    "            print(\".\", end=\"\", flush=True)\n",
    "            time.sleep(60)\n",
    "            schedule_desc = model_monitor.describe_schedule()\n",
    "            execution_summary = schedule_desc[\"LastMonitoringExecutionSummary\"]\n",
    "        print()\n",
    "        print(f\"Done! Execution Status: {execution_summary['MonitoringExecutionStatus']}\")\n",
    "    else:\n",
    "        print(\"Last execution not found\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: The following cell takes about 10 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "wait_for_execution_to_finish(model_bias_monitor)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Merged data\n",
    "\n",
    "Merged data is the intermediate results of bias drift monitoring execution. It is saved to JSON Lines files under the \"merge\" folder of `monitor_output_s3_uri`. Each line is a valid JSON object which combines the captured data and the ground truth data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "merged_data_s3_uri = f\"{monitor_output_s3_uri}/merge\"\n",
    "merged_data_files = sorted(\n",
    "    sagemaker.s3.S3Downloader.list(\n",
    "        s3_uri=merged_data_s3_uri,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    ")\n",
    "print(\"Found merged data files:\")\n",
    "print(\"\\n \".join(merged_data_files[-5:]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell prints a single line of a merged data file.\n",
    "\n",
    "* `eventId` is the inference ID from the captured data and the ground truth data\n",
    "* `groundTruthData` is from the ground truth data\n",
    "* `captureData` is from the captured data. In this case, the `data` of `batchTransformOutput` is from the transform output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "merged_record = sagemaker.s3.S3Downloader.read_file(\n",
    "    s3_uri=merged_data_files[-1],\n",
    "    sagemaker_session=sagemaker_session,\n",
    ").splitlines()[0]\n",
    "print(json.dumps(json.loads(merged_record), indent=4))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspect execution results\n",
    "\n",
    "List the generated reports,\n",
    "\n",
    "* analysis.json includes all the bias metrics.\n",
    "* report.* files are static report files to visualize the bias metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "schedule_desc = model_bias_monitor.describe_schedule()\n",
    "execution_summary = schedule_desc.get(\"LastMonitoringExecutionSummary\")\n",
    "if execution_summary and execution_summary[\"MonitoringExecutionStatus\"] in [\n",
    "    \"Completed\",\n",
    "    \"CompletedWithViolations\",\n",
    "]:\n",
    "    last_model_bias_monitor_execution = model_bias_monitor.list_executions()[-1]\n",
    "    last_model_bias_monitor_execution_report_uri = (\n",
    "        last_model_bias_monitor_execution.output.destination\n",
    "    )\n",
    "    print(f\"Report URI: {last_model_bias_monitor_execution_report_uri}\")\n",
    "    last_model_bias_monitor_execution_report_files = sorted(\n",
    "        sagemaker.s3.S3Downloader.list(\n",
    "            s3_uri=last_model_bias_monitor_execution_report_uri,\n",
    "            sagemaker_session=sagemaker_session,\n",
    "        )\n",
    "    )\n",
    "    print(\"Found Report Files:\")\n",
    "    print(\"\\n \".join(last_model_bias_monitor_execution_report_files))\n",
    "else:\n",
    "    last_model_bias_monitor_execution = None\n",
    "    print(\n",
    "        \"====STOP==== \\n No completed executions to inspect further. Please wait till an execution completes or investigate previously reported failures.\"\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are any violations compared to the baseline, they are listed here. See [Bias Drift Violations](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-bias-drift-violations.html) for the schema of the file, and how violations are detected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "violations = model_bias_monitor.latest_monitoring_constraint_violations()\n",
    "if violations is not None:\n",
    "    pprint.PrettyPrinter(indent=4).pprint(violations.body_dict)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the analysis results are also published to CloudWatch, see [CloudWatch Metrics for Bias Drift Analysis](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-bias-drift-cw.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "If there is no plan to collect more data for bias drift monitoring, then the monitor should be stopped (and deleted) to avoid incurring additional charges. Note that deleting the monitor does not delete the data in S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bias_monitor.stop_monitoring_schedule()\n",
    "wait_for_execution_to_finish(model_bias_monitor)\n",
    "model_bias_monitor.delete_monitoring_schedule()\n",
    "sagemaker_session.delete_model(model_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker_model_monitor|fairness_and_explainability|SageMaker-Monitoring-Bias-Drift-for-Batch-Transform.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}