 archived/autopilot_ts_data_merge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2c2f421d-217a-446e-98a7-5d0d8c4225d1",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# Time-Series Forecasting - Merge Amazon Forecast Datasets for Amazon SageMaker Canvas API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4889982e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b01027dc",
   "metadata": {},
   "source": [
    "### 1. Introduction <a name='introduction'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc0c305-d003-41d4-98dd-98262d9ac778",
   "metadata": {
    "tags": []
   },
   "source": [
    "Canvas API for TimeSeries Forecasting uses one dataset unlike Amazon Forecast which has seperate datasets for target-time-series(tts), related-time-series(rts) and item-meta-data. This notebook is useful for customer who would like to move from Amazon Forecast to SageMaker Canvas as it demonstrates 1/ using python code snippet to combine 3 different datasets into one, 2/create a configuration file to separately identify related time series field from item-metadata, and 3/uses autoML API to progrmatically train the dataset and make batch inference.\n",
    "\n",
    "These artifacts include: \n",
    "- backtest (holdout) forecasts per base model over multiple time windows,\n",
    "- accuracy metrics per base model,\n",
    "- backtest results and accuracy metrics for the ensembled model,\n",
    "- a scaled explainability report displaying the importance of each covariate and static metadata feature.\n",
    "- all model artifacts are provided as well on S3, which can be registered or use for batch/real-time inference\n",
    "\n",
    "If you are running this notebook in your own environment, you can bring in your own data. The sample dataset we have used is available [here](https://amazon-forecast-samples.s3.us-west-2.amazonaws.com/ml_ops/FoodDemand.zip). Once you download the data set, unzip and take out the three csv files, create a 'data' folder in the same level with the notebook and place the csv files in the 'data' folder."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "806be3a5-97c1-4e1a-9eef-366d78ae459f",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 2. Setup <a name='setup'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e57f6e-3545-4bc8-bf01-f14263a71c7d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Update boto3 using this method, or your preferred method\n",
    "!pip install --upgrade boto3 --quiet\n",
    "!pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec5d60d7-2760-4866-a37c-37bcd1fda64e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is the client we will use to interact with SageMaker Autopilot\n",
    "import sagemaker\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "import os\n",
    "import json\n",
    "from sagemaker import get_execution_role\n",
    "from time import gmtime, strftime, sleep\n",
    "import pandas as pd\n",
    "from datetime import datetime as dt\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "session = sagemaker.Session()\n",
    "client = boto3.client(\"sts\")\n",
    "account_id = client.get_caller_identity()[\"Account\"]\n",
    "\n",
    "# Modify the following default_bucket to use a bucket of your choosing\n",
    "bucket = session.default_bucket()\n",
    "data_bucket = \"rawdata-\" + region + \"-\" + account_id\n",
    "# bucket = 'my-bucket'\n",
    "prefix = \"moving-to-canvas\"\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "# This is the client we will use to interact with SageMaker Autopilot\n",
    "sm = boto3.Session().client(service_name=\"sagemaker\", region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba114977-9b33-4490-83dc-cc23e6ae528c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Assign column heading to 3 different data files for target time series, related time series and item metadata\n",
    "columns_tts = [\"item_id\", \"store_id\", \"demand\", \"ts\"]\n",
    "\n",
    "columns_rts = [\"item_id\", \"store_id\", \"price\", \"ts\"]\n",
    "\n",
    "columns_items = [\"item_id\", \"item_type\", \"item_description\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "131715d5-1870-44a3-8929-7a08c2f0ba52",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Read from data file and explore the data. Also change the time stamp format to desired one if needed.\n",
    "tbl_tts = pd.read_csv(\"data/food-forecast-tts-uc1.csv\", header=None)\n",
    "tbl_tts.columns = columns_tts\n",
    "tbl_tts[\"ts\"] = pd.to_datetime(tbl_tts[\"ts\"], format=\"%m/%d/%y\").dt.strftime(\"%Y-%m-%d\")\n",
    "# print(tbl_tts.shape)\n",
    "# tbl_tts.head()\n",
    "# tbl_tts['ts'].min(), tbl_tts['ts'].max()\n",
    "# print(tbl_tts.dtypes)\n",
    "# print(tbl_tts.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e4d9513-fdc9-475f-b010-ef4bd396fca9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# read from data file and explore the data. Also change the time stamp format to desired one if needed.\n",
    "tbl_rts = pd.read_csv(\"data/food-forecast-rts-uc1.csv\", header=None)\n",
    "tbl_rts.columns = columns_rts\n",
    "tbl_rts[\"ts\"] = pd.to_datetime(tbl_rts[\"ts\"], format=\"%m/%d/%y\").dt.strftime(\"%Y-%m-%d\")\n",
    "# print(tbl_rts.shape)\n",
    "# tbl_rts .head()\n",
    "# tbl_rts['ts'].min(), tbl_rts['ts'].max()\n",
    "# print(tbl_rts.dtypes)\n",
    "# print(tbl_rts.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4604d30-f974-4076-b8c4-96ca554fd7db",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# read from data file and explore the data\n",
    "tbl_item = pd.read_csv(\"data/food-forecast-item.csv\", header=None)\n",
    "tbl_item.columns = columns_items\n",
    "# tbl_item = tbl_item.set_index('item_id', inplace=True)\n",
    "# print(tbl_item.shape)\n",
    "# tbl_item.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a64be31-3cfd-48a2-88f7-7e2e15c15e15",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Join the data files into one data file\n",
    "tts_rts_combined_outer = tbl_tts.merge(tbl_rts, how=\"outer\")\n",
    "tts_rts_combined_outer\n",
    "combined_tts_rts_im = tts_rts_combined_outer.merge(tbl_item, how=\"left\")\n",
    "combined_tts_rts_im"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "173891f8-4082-4916-8ef4-199aeb62086a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Write the combined dataset to csv file which will be used for training the model using SageMaker Canvas API\n",
    "file_name = \"combined_tts_rts_item.csv\"\n",
    "full_path = \"data/\" + file_name\n",
    "combined_tts_rts_im.to_csv(full_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b63134a-8615-4152-a120-2ae57a9618fa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# All columns in tts will be included in TimeSeriesConfig as it contains\n",
    "# target, itemID, timestamp, and additional forecast dimensions.\n",
    "exclude_columns = columns_tts\n",
    "columns_to_include = [col for col in combined_tts_rts_im.columns if col not in exclude_columns]\n",
    "\n",
    "json_data = {\"FeatureAttributeNames\": columns_to_include, \"FeatureDataTypes\": {}}\n",
    "\n",
    "for col in columns_to_include:\n",
    "    dtype = combined_tts_rts_im[col].dtype\n",
    "    # All rts columns must be numeric to be treated as related features\n",
    "    if col in columns_rts:\n",
    "        json_data[\"FeatureDataTypes\"][col] = \"numeric\"\n",
    "    elif isinstance(dtype, pd.CategoricalDtype):\n",
    "        json_data[\"FeatureDataTypes\"][col] = \"categorical\"\n",
    "    elif pd.api.types.is_datetime64_any_dtype(dtype):\n",
    "        json_data[\"FeatureDataTypes\"][col] = \"datetime\"\n",
    "    else:\n",
    "        json_data[\"FeatureDataTypes\"][col] = \"text\"\n",
    "\n",
    "json_str = json.dumps(json_data, indent=4)\n",
    "\n",
    "# print(json_str)\n",
    "\n",
    "with open(\"data/feature.json\", \"w\") as f:\n",
    "    f.write(json_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8144e32-27f2-4c74-9ac8-7744670d51f3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Upload the data file and config file to S3 bucket\n",
    "\n",
    "s3 = boto3.client(\"s3\")\n",
    "object_name = prefix + \"/train/\" + file_name\n",
    "# print(object_name)\n",
    "try:\n",
    "    response = s3.upload_file(full_path, bucket, object_name)\n",
    "except ClientError as e:\n",
    "    logging.error(e)\n",
    "\n",
    "config_file_name = \"feature.json\"\n",
    "object_name = prefix + \"/\" + config_file_name\n",
    "config_full_path = \"data/\" + config_file_name\n",
    "\n",
    "try:\n",
    "    response = s3.upload_file(config_full_path, data_bucket, object_name)\n",
    "except ClientError as e:\n",
    "    logging.error(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "109f5f2b-a15e-447a-8990-387943a5eaf0",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3. Model Training <a name='training'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4547b63f-4181-4ac6-9b13-5814d1195cb8",
   "metadata": {},
   "source": [
    "Establish an AutoML training job name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "120446de-08a9-48be-9c7a-afc7d7e6dbce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "timestamp_suffix = strftime(\"%Y%m%d-%H%M%S\", gmtime())\n",
    "auto_ml_job_name = \"ts-\" + timestamp_suffix\n",
    "print(\"AutoMLJobName: \" + auto_ml_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843ada35-f6a5-42e1-b7ca-f14e127661a5",
   "metadata": {},
   "source": [
    "Define training job specifications. More information about [create_auto_ml_job_v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) can be found in our SageMaker documentation.</n></n>This JSON body leverages the built-in sample data schema. Please consult the documentation to understand how to alter the parameters for your unique schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5756d7d0-c2fe-4aed-9c52-0dffc1400fdc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "input_data_config = [\n",
    "    {\n",
    "        \"ChannelType\": \"training\",\n",
    "        \"ContentType\": \"text/csv;header=present\",\n",
    "        \"CompressionType\": \"None\",\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": \"s3://{}/{}/train/\".format(bucket, prefix),\n",
    "            }\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "output_data_config = {\"S3OutputPath\": \"s3://{}/{}/train_output\".format(bucket, prefix)}\n",
    "\n",
    "optimizaton_metric_config = {\"MetricName\": \"AverageWeightedQuantileLoss\"}\n",
    "\n",
    "automl_problem_type_config = {\n",
    "    \"TimeSeriesForecastingJobConfig\": {\n",
    "        \"FeatureSpecificationS3Uri\": \"s3://{}/{}/feature.json\".format(data_bucket, prefix),\n",
    "        \"ForecastFrequency\": \"M\",\n",
    "        \"ForecastHorizon\": 2,\n",
    "        \"ForecastQuantiles\": [\"p50\", \"p60\", \"p70\", \"p80\", \"p90\"],\n",
    "        \"Transformations\": {\n",
    "            \"Filling\": {\n",
    "                \"demand\": {\"middlefill\": \"zero\", \"backfill\": \"zero\"},\n",
    "                \"price\": {\"middlefill\": \"zero\", \"backfill\": \"zero\", \"futurefill\": \"zero\"},\n",
    "            }\n",
    "        },\n",
    "        \"TimeSeriesConfig\": {\n",
    "            \"TargetAttributeName\": \"demand\",\n",
    "            \"TimestampAttributeName\": \"ts\",\n",
    "            \"ItemIdentifierAttributeName\": \"item_id\",\n",
    "            \"GroupingAttributeNames\": [\"store_id\"],\n",
    "        },\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "869710c7-6474-4726-a06f-b1e7cd1bdd4c",
   "metadata": {},
   "source": [
    "With parameters now defined, invoke the [training job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) and monitor for its completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ae4865-32c5-4079-83a3-e5177fdf5784",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sm.create_auto_ml_job_v2(\n",
    "    AutoMLJobName=auto_ml_job_name,\n",
    "    AutoMLJobInputDataConfig=input_data_config,\n",
    "    OutputDataConfig=output_data_config,\n",
    "    AutoMLProblemTypeConfig=automl_problem_type_config,\n",
    "    AutoMLJobObjective=optimizaton_metric_config,\n",
    "    RoleArn=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb8ae02-2b67-4fe6-b3d5-c258dd625671",
   "metadata": {},
   "source": [
    "Next, we demonstrate a looping mechanism to query (monitor) job status. When the status is ```Completed```, you may review the accuracy of the model and decide whether to perform inference on a batch or real-time API basis as described in this notebook. Please consult documentation for [describe_auto_ml_job_v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/describe_auto_ml_job_v2.html) as needed.\n",
    "\n",
    "NOTE: Training the Model will take approxmiately 30 minutes. Please take this time to work other Labs in this workshop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e9b0f88-364b-4263-8d0a-589039c28f3d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "describe_response = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)\n",
    "job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"Completed\", \"Stopped\"):\n",
    "    describe_response = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)\n",
    "    job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "    print(\n",
    "        dt.now(),\n",
    "        describe_response[\"AutoMLJobStatus\"]\n",
    "        + \" - \"\n",
    "        + describe_response[\"AutoMLJobSecondaryStatus\"],\n",
    "    )\n",
    "    sleep(180)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42851162-fdc3-4eed-90b3-b03e2879de46",
   "metadata": {},
   "source": [
    "Once training is completed, you can use the describe function to iterate over model leaderboard results. Below is an example to use the best candidate in the subsequent inference phase. Please consult our documentation on [create_model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_model.html) as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4a37ec7-9125-412f-9b12-194cee807096",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "best_candidate = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)[\"BestCandidate\"]\n",
    "best_candidate_containers = best_candidate[\"InferenceContainers\"]\n",
    "best_candidate_name = best_candidate[\"CandidateName\"]\n",
    "\n",
    "reponse = sm.create_model(\n",
    "    ModelName=best_candidate_name, ExecutionRoleArn=role, Containers=best_candidate_containers\n",
    ")\n",
    "\n",
    "print(\"BestCandidateName:\", best_candidate_name)\n",
    "print(\"BestCandidateContainers:\", best_candidate_containers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "526ca95f-36b8-48ff-87f0-a3027792d0e5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### 4. Batch Predictions (Inference) <a name='batch'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "674870e5-683c-4350-ae80-6fe6dc03d664",
   "metadata": {},
   "source": [
    "Please review [service limits](https://docs.aws.amazon.com/marketplace/latest/userguide/ml-service-restrictions-and-limits.html\n",
    ") with batch transform. At the time of writing, the documentation says the maximum size of the input data per invocation is 100 MB. Translated, when working with \n",
    "datasets over 100MB, you will need to prepare your data by splitting/sharding into multiple files.\n",
    " Take care to ensure each file contains whole time series. One potential way to do this is to use\n",
    " a function that splits data on the item key, or similar.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a69661d-4849-4fe7-9254-2b773660a580",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "timestamp_suffix = strftime(\"%Y%m%d-%H%M%S\", gmtime())\n",
    "transform_job_name = f\"{best_candidate_name}-\" + timestamp_suffix\n",
    "print(\"BatchTransformJob: \" + transform_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8dde0b-b01f-4659-a75b-27a494ecccf8",
   "metadata": {
    "tags": []
   },
   "source": [
    "The next cell downloads a dataset once again and this time places in a ```batch_transform/input``` folder. Ideally, this input dataset can be all of your time-series, or a fraction thereof. Please take care to ensure the dataset is within the limits described."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b64068a-a790-454e-8102-39a0a4dfd0db",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# modify the input file for inference to remove n/a values\n",
    "df = pd.read_csv(\"data/combined_tts_rts_item.csv\")\n",
    "df.fillna(0, inplace=True)\n",
    "df.to_csv(\"data/combined_tts_rts_item_modified.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a23fdc9b-e176-4e61-9675-80ecd43a5652",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Upload the data file to S3 bucket for batch prediction\n",
    "s3 = boto3.client(\"s3\")\n",
    "file_name = \"combined_tts_rts_item.csv\"\n",
    "modified_file_name = \"combined_tts_rts_item_modified.csv\"\n",
    "full_path = \"data/\" + modified_file_name\n",
    "object_name = prefix + \"/batch_transform/input/\" + file_name\n",
    "# print(object_name)\n",
    "try:\n",
    "    response = s3.upload_file(full_path, bucket, object_name)\n",
    "except ClientError as e:\n",
    "    logging.error(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6347f1c2-7c7e-45ac-aeda-0ddc9826fd33",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = sm.create_transform_job(\n",
    "    TransformJobName=transform_job_name,\n",
    "    ModelName=best_candidate_name,\n",
    "    MaxPayloadInMB=0,\n",
    "    ModelClientConfig={\"InvocationsTimeoutInSeconds\": 3600},\n",
    "    TransformInput={\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": \"s3://{}/{}/batch_transform/input/\".format(bucket, prefix),\n",
    "            }\n",
    "        },\n",
    "        \"ContentType\": \"text/csv\",\n",
    "        \"SplitType\": \"None\",\n",
    "    },\n",
    "    TransformOutput={\n",
    "        \"S3OutputPath\": \"s3://{}/{}/batch_transform/output/\".format(bucket, prefix),\n",
    "        \"AssembleWith\": \"Line\",\n",
    "    },\n",
    "    TransformResources={\"InstanceType\": \"ml.m5.4xlarge\", \"InstanceCount\": 1},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51fe8198-ed78-4e4c-83d4-4da6faf172cd",
   "metadata": {},
   "source": [
    "Poll for batch transformation job to complete. Once completed, resulting prediction files are available at the URI shown in the prior cell, ```S3OutputPath```. We use the API method [describe_transform_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/describe_transform_job.html) to complete this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67e4132f-bbda-4e73-a59a-65da70bad5b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "describe_response = sm.describe_transform_job(TransformJobName=transform_job_name)\n",
    "\n",
    "job_run_status = describe_response[\"TransformJobStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"Completed\", \"Stopped\"):\n",
    "    describe_response = sm.describe_transform_job(TransformJobName=transform_job_name)\n",
    "    job_run_status = describe_response[\"TransformJobStatus\"]\n",
    "\n",
    "    print(dt.now(), describe_response[\"TransformJobStatus\"])\n",
    "    sleep(120)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1928ccbb-2ee5-4f53-8d2f-0125e9c3bb80",
   "metadata": {
    "tags": []
   },
   "source": [
    "Once the batch predictions are complete, download and review the resulting output. This will display the first 10 predictions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df9142c0-f527-4959-9714-37e0d3fbeb4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3 = boto3.resource(\"s3\")\n",
    "s3.Bucket(bucket).download_file(\n",
    "    \"{}/batch_transform/output/combined_tts_rts_item.csv.out\".format(prefix),\n",
    "    \"combined_tts_rts_item.csv.out\",\n",
    ")\n",
    "df = pd.read_csv(\"combined_tts_rts_item.csv.out\")\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0005ad58",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/autopilot|autopilot_ts_data_merge.ipynb\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/autopilot|autopilot_ts_data_merge.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/autopilot|autopilot_ts_data_merge.ipynb)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}