        end_to_end_ml_lifecycle/sm-autopilot_time_series_forecasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Time-Series Forecasting with Amazon SageMaker Autopilot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74caeb30",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contents\n",
    "\n",
    "1. [Introduction](#introduction)\n",
    "1. [Setup](#setup)\n",
    "1. [Model Training](#training)\n",
    "1. [Real-Time Predictions (Inference)](#realtime)\n",
    "1. [Batch Predictions (Inference)](#batch)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1. Introduction <a name='introduction'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses Amazon SageMaker Autopilot to train a time-series model and produce predictions against the trained model. At the top-level, customers provide a set of tabular historical data on S3 and make an API to train a model. Once the model has been trained, you can elect to produce prediction as a batch or via a real-time endpoint.</n></n>  As part of the training process, SageMaker Autopilot manages and runs multiple time series models concurrently. All of these models are combined into a single ensembled model which blends the candidate models in a ratio that minimizes forecast error. Customers are provided with metadata and models for the ensemble and all underlying candidate models too. SageMaker Autopilot orchestrates this entire process and provides several artifacts as a result.\n",
    "\n",
    "These artifacts include: \n",
    "- backtest (holdout) forecasts per base model over multiple time windows,\n",
    "- accuracy metrics per base model,\n",
    "- backtest results and accuracy metrics for the ensembled model,\n",
    "- a scaled explainability report displaying the importance of each covariate and static metadata feature.\n",
    "- all model artifacts are provided as well on S3, which can be registered or use for batch/real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Setup <a name='setup'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update boto3 using this method, or your preferred method\n",
    "!pip install --upgrade boto3 --quiet\n",
    "!pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from sagemaker import get_execution_role\n",
    "from time import gmtime, strftime, sleep\n",
    "import datetime\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "session = sagemaker.Session()\n",
    "\n",
    "# Modify the following default_bucket to use a bucket of your choosing\n",
    "bucket = session.default_bucket()\n",
    "# bucket = 'my-bucket'\n",
    "prefix = \"my-project-name\"\n",
    "default_bucket_prefix = session.default_bucket_prefix\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "# This is the client we will use to interact with SageMaker Autopilot\n",
    "sm = boto3.Session().client(service_name=\"sagemaker\", region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide a sample set of data to accompany this notebook. You may use our synthetic dataset, or alter this notebook to accommodate your own data. As a note, the next cell will copy a file to your S3 bucket and prefix defined in the last cell. As an alternate, we provide a method to copy the file to your local disk too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT: When training a model, your input data can contain a mixture of covariate and static item metadata. Take care to create future-dated rows that extend to the end of your prediction horizon. In the future-dated rows, carry your static item metadata and expected covariate values. Future-dated target-value (y) should be empty. Please download the example synthetic file using the S3 copy command in the next cell. You can observe the data programmatically or in a text editor as an example.\n",
    "\n",
    "The structure of the CSV file provided is as follows:\n",
    "- product_code (required: ItemIdentifierAttributeName)\n",
    "- product_category (static, categorical feature describing product_code)\n",
    "- product_subcategory (static, categorical feature describing product_code)\n",
    "- location_code (GroupingAttributeNames column to get predictions at product_code + location_code)\n",
    "- scaled_price (covariate)\n",
    "- promotion_email (covariate)\n",
    "- promotion_homepage (covariate)\n",
    "- timestamp (required, TimestampAttributeName)\n",
    "- unit_sales (required: TargetAttributeName)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource(\"s3\")\n",
    "copy_source = {\"Bucket\": \"amazon-forecast-samples\", \"Key\": \"autopilot/synthetic-food-demand.csv\"}\n",
    "\n",
    "s3.meta.client.copy(copy_source, bucket, prefix + \"/train/synthetic-food-demand.csv\")\n",
    "\n",
    "# Alternate, copy the file for local inspection/use.\n",
    "#!aws s3 cp s3://amazon-forecast-samples/autopilot/synthetic-food-demand.csv ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Model Training <a name='training'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Establish an AutoML training job name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp_suffix = strftime(\"%Y%m%d-%H%M%S\", gmtime())\n",
    "auto_ml_job_name = \"ts-\" + timestamp_suffix\n",
    "print(\"AutoMLJobName: \" + auto_ml_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define training job specifications. More information about [create_auto_ml_job_v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) can be found in our SageMaker documentation.</n></n>This JSON body leverages the built-in sample data schema. Please consult the documentation to understand how to alter the parameters for your unique schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_config = [\n",
    "    {\n",
    "        \"ChannelType\": \"training\",\n",
    "        \"ContentType\": \"text/csv;header=present\",\n",
    "        \"CompressionType\": \"None\",\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": \"s3://{}/{}/train/\".format(bucket, prefix),\n",
    "            }\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "output_data_config = {\"S3OutputPath\": \"s3://{}/{}/train_output\".format(bucket, prefix)}\n",
    "\n",
    "optimizaton_metric_config = {\"MetricName\": \"AverageWeightedQuantileLoss\"}\n",
    "\n",
    "automl_problem_type_config = {\n",
    "    \"TimeSeriesForecastingJobConfig\": {\n",
    "        \"ForecastFrequency\": \"W\",\n",
    "        \"ForecastHorizon\": 4,\n",
    "        \"ForecastQuantiles\": [\"p50\", \"p60\", \"p70\", \"p80\", \"p90\"],\n",
    "        \"Transformations\": {\n",
    "            \"Filling\": {\n",
    "                \"unit_sales\": {\"middlefill\": \"zero\", \"backfill\": \"zero\"},\n",
    "                \"promotion_email\": {\"middlefill\": \"zero\", \"backfill\": \"zero\", \"futurefill\": \"zero\"},\n",
    "                \"promotion_homepage\": {\n",
    "                    \"middlefill\": \"zero\",\n",
    "                    \"backfill\": \"zero\",\n",
    "                    \"futurefill\": \"zero\",\n",
    "                },\n",
    "                \"promotion_email\": {\"middlefill\": \"zero\", \"backfill\": \"zero\", \"futurefill\": \"zero\"},\n",
    "                \"scaled_price\": {\n",
    "                    \"middlefill\": \"value\",\n",
    "                    \"middlefill_value\": \"1\",\n",
    "                    \"backfill\": \"value\",\n",
    "                    \"backfill_value\": \"1\",\n",
    "                    \"futurefill\": \"value\",\n",
    "                    \"futurefill_value\": \"1\",\n",
    "                },\n",
    "            }\n",
    "        },\n",
    "        \"TimeSeriesConfig\": {\n",
    "            \"TargetAttributeName\": \"unit_sales\",\n",
    "            \"TimestampAttributeName\": \"timestamp\",\n",
    "            \"ItemIdentifierAttributeName\": \"product_code\",\n",
    "            \"GroupingAttributeNames\": [\"location_code\"],\n",
    "        },\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With parameters now defined, invoke the [training job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) and monitor for its completion. You can expect the training to take about 1 hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.create_auto_ml_job_v2(\n",
    "    AutoMLJobName=auto_ml_job_name,\n",
    "    AutoMLJobInputDataConfig=input_data_config,\n",
    "    OutputDataConfig=output_data_config,\n",
    "    AutoMLProblemTypeConfig=automl_problem_type_config,\n",
    "    AutoMLJobObjective=optimizaton_metric_config,\n",
    "    RoleArn=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we demonstrate a looping mechanism to query (monitor) job status. When the status is ```Completed```, you may review the accuracy of the model and decide whether to perform inference on a batch or real-time API basis as described in this notebook. Please consult documentation for [describe_auto_ml_job_v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/describe_auto_ml_job_v2.html) as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "describe_response = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)\n",
    "job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"Completed\", \"Stopped\"):\n",
    "    describe_response = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)\n",
    "    job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "    print(\n",
    "        datetime.datetime.now(),\n",
    "        describe_response[\"AutoMLJobStatus\"]\n",
    "        + \" - \"\n",
    "        + describe_response[\"AutoMLJobSecondaryStatus\"],\n",
    "    )\n",
    "    sleep(180)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once training is completed, you can use the describe function to iterate over model leaderboard results. Below is an example to use the best candidate in the subsequent inference phase. Please consult our documentation on [create_model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_model.html) as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_candidate = sm.describe_auto_ml_job_v2(AutoMLJobName=auto_ml_job_name)[\"BestCandidate\"]\n",
    "best_candidate_containers = best_candidate[\"InferenceContainers\"]\n",
    "best_candidate_name = best_candidate[\"CandidateName\"]\n",
    "\n",
    "reponse = sm.create_model(\n",
    "    ModelName=best_candidate_name, ExecutionRoleArn=role, Containers=best_candidate_containers\n",
    ")\n",
    "\n",
    "print(\"BestCandidateName:\", best_candidate_name)\n",
    "print(\"BestCandidateContainers:\", best_candidate_containers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Real-Time Predictions (Inference) <a name='realtime'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to perform real-time inference, review this section. If you want to perform batch processing, you may skip the real-time inference section and move to [Batch Predictions (Inference)](#batch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a model, endpoint configuration and endpoint name using the candidate metadata. Consult [create_endpoint_config](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_endpoint_config.html) documentation for more detail. Additionally, please adjust the ```InstanceType``` and ```InitialInstanceCount``` according to need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT: The data you supply for inference must have at least four valid historical values for each time-series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_name = f\"epc-{best_candidate_name}\"\n",
    "endpoint_name = f\"ep-{best_candidate_name}\"\n",
    "\n",
    "production_variants = [\n",
    "    {\n",
    "        \"InstanceType\": \"ml.m5.2xlarge\",\n",
    "        \"InitialInstanceCount\": 1,\n",
    "        \"ModelName\": best_candidate_name,\n",
    "        \"VariantName\": \"AllTraffic\",\n",
    "    }\n",
    "]\n",
    "\n",
    "epc_response = sm.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name, ProductionVariants=production_variants\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you can deploy a real-time endpoint using the [create_endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_endpoint.html\n",
    ") API. See the documentation for more details and options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Poll for endpoint to become ready to serve (InService)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "describe_response = sm.describe_endpoint(EndpointName=endpoint_name)\n",
    "\n",
    "job_run_status = describe_response[\"EndpointStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"InService\", \"Stopped\"):\n",
    "    describe_response = sm.describe_endpoint(EndpointName=endpoint_name)\n",
    "    job_run_status = describe_response[\"EndpointStatus\"]\n",
    "\n",
    "    print(datetime.datetime.now(), describe_response[\"EndpointStatus\"])\n",
    "    sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cells help demonstrate opening a local CSV file for inference. Alternately, this data could come from S3, a database query or live application. In this example, the data is loaded into a Python memory object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A small sample file that corresponds to the sample training dataset and trained model schema\n",
    "!aws s3 cp s3://amazon-forecast-samples/autopilot/real-time-payload.csv ./real-time-payload.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_file = \"./real-time-payload.csv\"\n",
    "f = open(input_file, \"r\")\n",
    "inference_data = f.read()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Method to instantiate SageMaker runtime client and [invoke endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime/client/invoke_endpoint.html). Please note the guidance given in the documentation page which says the response must be within 60 seconds of invocation. Response times are a function of payload size. You should take care to provide history for single time-series, and carefully testing the ability to perform predictions for more than one time-series at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_client = boto3.client(\"sagemaker-runtime\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name, Body=inference_data, ContentType=\"text/csv\"\n",
    ")\n",
    "\n",
    "prediction = response[\"Body\"].read().decode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the results from the real-time API call are loaded into a variable called ```prediction```. You have several options on what you do with the results. A few are given here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional: Example of saving the resulting real-time predictions to a local filesystem. Carefully plan your naming convention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file = \"real-time-prediction-output.csv\"\n",
    "f = open(output_file, \"w\")\n",
    "f.write(prediction)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional: Example of saving the resulting real-time predictions to a S3 object. Carefully plan your naming convention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file = \"real-time-prediction-output.csv\"\n",
    "key = \"s3://{}/{}/realtime_inference/{}\".format(bucket, prefix, output_file)\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "s3_client.put_object(Body=prediction, Bucket=bucket, Key=key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional: Example of loading predictions into a Panda dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from io import StringIO\n",
    "\n",
    "df = pd.read_csv(StringIO(prediction), sep=\",\")\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you may also elect to place predictions in a SQS queue, post them to a stream, or post them to another real-time API according to need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cleanup Real-time Endpoint Resources\n",
    "\n",
    "As needed, you can stop the endpoint and related billing costs as follows. When you need the endpoint again, you can follow the deployment steps again. Ideally, at a future time, another newer model is trained and able to be deployed as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.delete_endpoint(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Caution: Do not delete the model if you intend on testing batch transformation too. If you do delete the model, you may redeploy it as long as the model artifact exists on S3. The next cell provides the container and S3 location for your best candidate model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"BestCandidateContainers:\", best_candidate_containers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.delete_model(ModelName=best_candidate_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Batch Predictions (Inference) <a name='batch'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please review [service limits](https://docs.aws.amazon.com/marketplace/latest/userguide/ml-service-restrictions-and-limits.html\n",
    ") with batch transform. At the time of writing, the documentation says the maximum size of the input data per invocation is 100 MB. Translated, when working with \n",
    "datasets over 100MB, you will need to prepare your data by splitting/sharding into multiple files.\n",
    " Take care to ensure each file contains whole time series. One potential way to do this is to use\n",
    " a function that splits data on the item key, or similar.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch Batch Transformation Job using [create_transform_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_transform_job.html). The runtime of the job is a function of the size of your data and the InstanceType and InstanceCount provided. Once the task is complete, results are available on S3 at the declared ```S3OutputPath``` location. From there, you can use an event handler or other mechanism to consume the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp_suffix = strftime(\"%Y%m%d-%H%M%S\", gmtime())\n",
    "transform_job_name = f\"{best_candidate_name}-\" + timestamp_suffix\n",
    "print(\"BatchTransformJob: \" + transform_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell downloads a dataset once again and this time places in a ```batch_transform/input``` folder. Ideally, this input dataset can be all of your time-series, or a fraction thereof. Please take care to ensure the dataset is within the limits described."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT: The data you supply for inference must have at least four valid historical values for each time-series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource(\"s3\")\n",
    "copy_source = {\"Bucket\": \"amazon-forecast-samples\", \"Key\": \"autopilot/synthetic-food-demand.csv\"}\n",
    "\n",
    "s3.meta.client.copy(\n",
    "    copy_source, bucket, prefix + \"/batch_transform/input/synthetic-food-demand.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sm.create_transform_job(\n",
    "    TransformJobName=transform_job_name,\n",
    "    ModelName=best_candidate_name,\n",
    "    MaxPayloadInMB=0,\n",
    "    ModelClientConfig={\"InvocationsTimeoutInSeconds\": 3600},\n",
    "    TransformInput={\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": \"s3://{}/{}/batch_transform/input/\".format(bucket, prefix),\n",
    "            }\n",
    "        },\n",
    "        \"ContentType\": \"text/csv\",\n",
    "        \"SplitType\": \"None\",\n",
    "    },\n",
    "    TransformOutput={\n",
    "        \"S3OutputPath\": \"s3://{}/{}/batch_transform/output/\".format(bucket, prefix),\n",
    "        \"AssembleWith\": \"Line\",\n",
    "    },\n",
    "    TransformResources={\"InstanceType\": \"ml.m5.12xlarge\", \"InstanceCount\": 1},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Poll for batch transformation job to complete. Once completed, resulting prediction files are available at the URI shown in the prior cell, ```S3OutputPath```. We use the API method [describe_transform_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/describe_transform_job.html) to complete this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "describe_response = sm.describe_transform_job(TransformJobName=transform_job_name)\n",
    "\n",
    "job_run_status = describe_response[\"TransformJobStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"Completed\", \"Stopped\"):\n",
    "    describe_response = sm.describe_transform_job(TransformJobName=transform_job_name)\n",
    "    job_run_status = describe_response[\"TransformJobStatus\"]\n",
    "\n",
    "    print(datetime.datetime.now(), describe_response[\"TransformJobStatus\"])\n",
    "    sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54cbc67b",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/end_to_end_ml_lifecycle|sm-autopilot_time_series_forecasting.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}