Object Detection/ObjectDetection_SageMaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying Bees Using Crowd Sourced Data using Amazon SageMaker \n",
    "\n",
    "### Table of contents\n",
    "1. [Introduction to dataset](#introduction)\n",
    "2. [Labeling with Amazon SageMaker Ground Truth](#groundtruth)\n",
    "3. [Reviewing labeling results](#review)\n",
    "4. [Training an Object Detection model](#training)\n",
    "5. [Review of Training Results](#review_training)\n",
    "6. [Model Tuning](#model_tuning)\n",
    "7. [Cleanup](#cleanup)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"introduction\"></a>\n",
    "## Introduction to dataset\n",
    "We will use a dataset from the [inaturalist.org](inaturalist.org) This dataset contains 500 images of bees that have been uploaded by inaturalist users for the purposes of recording the observation and identification. We only used images that their users have licensed under [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) license. For your convenience, we have placed the dataset in S3 in a single zip archive here: http://aws-tc-largeobjects.s3-us-west-2.amazonaws.com/DIG-TF-200-MLBEES-10-EN/dataset.zip\n",
    "\n",
    "First, download and unzip the archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget http://aws-tc-largeobjects.s3-us-west-2.amazonaws.com/DIG-TF-200-MLBEES-10-EN/dataset.zip \n",
    "!unzip -qo dataset.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The archive contains the following structure: 500 `.jpg` image files, a manifest file (to be explained later) and 10 test images in the `test` subfolder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -l dataset.zip | tail -20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's upload this dataset to your own S3 bucket in preparation for labeling and training using Amazon SageMaker. For this demo, we will be using `us-west-2` region, so your bucket needs to be in this region. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# S3 bucket must be created in us-west-2 (Oregon) region\n",
    "BUCKET = 'denisb-sagemaker-oregon'\n",
    "PREFIX = 'input' # this is the root path to your working space, feel to use a different path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 sync --exclude=\"*\" --include=\"[0-9]*.jpg\" . s3://$BUCKET/$PREFIX/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labeling with SageMaker Ground Truth <a name=\"groundtruth\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to run your first labeling with Amazon SageMaker Ground Truth. Follow the steps shown in the recording.\n",
    "\n",
    "When specifying information needed to configure the labeling UI tool, use the following information:\n",
    "\n",
    "- Brief task description: _\"Draw a bounding box around the bee in this image.\"_\n",
    "- Labels: _\"bee\"_\n",
    "- Good example description: _\"bounding box includes all visible parts of the insect - legs, antennae, etc.\"_\n",
    "- Good example image: https://s3.us-west-2.amazonaws.com/aws-tc-largeobjects/DIG-TF-200-MLBEES-10-EN/bee-good-5535715.jpg\n",
    "- Bad example description: _\"bounding box is too big and/or excludes some visible parts of the insect\"_\n",
    "- Bad example image: https://s3.us-west-2.amazonaws.com/aws-tc-largeobjects/DIG-TF-200-MLBEES-10-EN/bee-bad-5535715.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reviewing labeling results\n",
    "<a name=\"reviewing\"></a>\n",
    "\n",
    "After the labeling job has completed, we can see the results of image annotations right in the SageMaker console itself. The console displays each image as well as the bounding boxes around the bees that were drawn by human labelers.\n",
    "\n",
    "At the same time we can examine the results in the so-called augmented manifest file that was generated. Let's download and examine the manifest file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter the name of your job here\n",
    "labeling_job_name = 'bees'\n",
    "\n",
    "import boto3\n",
    "client = boto3.client('sagemaker')\n",
    "\n",
    "s3_output = client.describe_labeling_job(LabelingJobName=labeling_job_name)['OutputConfig']['S3OutputPath'] + labeling_job_name\n",
    "augmented_manifest_url = f'{s3_output}/manifests/output/output.manifest'\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "try:\n",
    "    os.makedirs('od_output_data/', exist_ok=False)\n",
    "except FileExistsError:\n",
    "    shutil.rmtree('od_output_data/')\n",
    "\n",
    "# now download the augmented manifest file and display first 3 lines\n",
    "!aws s3 cp $augmented_manifest_url od_output_data/\n",
    "augmented_manifest_file = 'od_output_data/output.manifest'\n",
    "!head -3 $augmented_manifest_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot all the annotated images. First, let's define a function that displays the local image file and draws over it the bounding boxes obtained via labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "from itertools import cycle\n",
    "\n",
    "def show_annotated_image(img_path, bboxes):\n",
    "    im = np.array(Image.open(img_path), dtype=np.uint8)\n",
    "    \n",
    "    # Create figure and axes\n",
    "    fig,ax = plt.subplots(1)\n",
    "\n",
    "    # Display the image\n",
    "    ax.imshow(im)\n",
    "\n",
    "    colors = cycle(['r', 'g', 'b', 'y', 'c', 'm', 'k', 'w'])\n",
    "    \n",
    "    for bbox in bboxes:\n",
    "        # Create a Rectangle patch\n",
    "        rect = patches.Rectangle((bbox['left'],bbox['top']),bbox['width'],bbox['height'],linewidth=1,edgecolor=next(colors),facecolor='none')\n",
    "\n",
    "        # Add the patch to the Axes\n",
    "        ax.add_patch(rect)\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, read the augmented manifest (JSON lines format) line by line and display the first 10 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip -q install --upgrade pip\n",
    "!pip -q install jsonlines\n",
    "import jsonlines\n",
    "from itertools import islice\n",
    "\n",
    "with jsonlines.open(augmented_manifest_file, 'r') as reader:\n",
    "    for desc in islice(reader, 10):\n",
    "        img_url = desc['source-ref']\n",
    "        img_file = os.path.basename(img_url)\n",
    "        file_exists = os.path.isfile(img_file)\n",
    "\n",
    "        bboxes = desc[labeling_job_name]['annotations']\n",
    "        show_annotated_image(img_file, bboxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='training'></a>\n",
    "## Training an Object Detection Model\n",
    "We are now ready to use the labeled dataset in order to train a Machine Learning model using the SageMaker [built-in Object Detection algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/object-detection.html).\n",
    "\n",
    "For this, we would need to split the full labeled dataset into a training and a validation datasets. Out of the total of 500 images we are going to use 400 for training and 100 for validation. The algorithm will use the first one to train the model and the latter to estimate the accuracy of the model, trained so far. The augmented manifest file from the previously run full labeling job was included in the original zip archive as `output.manifest`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with jsonlines.open('output.manifest', 'r') as reader:\n",
    "    lines = list(reader)\n",
    "    # Shuffle data in place.\n",
    "    np.random.shuffle(lines)\n",
    "    \n",
    "dataset_size = len(lines)\n",
    "num_training_samples = round(dataset_size*0.8)\n",
    "\n",
    "train_data = lines[:num_training_samples]\n",
    "validation_data = lines[num_training_samples:]\n",
    "\n",
    "augmented_manifest_filename_train = 'train.manifest'\n",
    "\n",
    "with open(augmented_manifest_filename_train, 'w') as f:\n",
    "    for line in train_data:\n",
    "        f.write(json.dumps(line))\n",
    "        f.write('\\n')\n",
    "\n",
    "augmented_manifest_filename_validation = 'validation.manifest'\n",
    "\n",
    "with open(augmented_manifest_filename_validation, 'w') as f:\n",
    "    for line in validation_data:\n",
    "        f.write(json.dumps(line))\n",
    "        f.write('\\n')\n",
    "        \n",
    "print(f'training samples: {num_training_samples}, validation samples: {len(lines)-num_training_samples}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's upload the two manifest files to S3 in preparation for training. We will use the same bucket you created earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pfx_training = PREFIX + '/training' if PREFIX else 'training'\n",
    "# Defines paths for use in the training job request.\n",
    "s3_train_data_path = 's3://{}/{}/{}'.format(BUCKET, pfx_training, augmented_manifest_filename_train)\n",
    "s3_validation_data_path = 's3://{}/{}/{}'.format(BUCKET, pfx_training, augmented_manifest_filename_validation)\n",
    "\n",
    "!aws s3 cp train.manifest s3://$BUCKET/$pfx_training/\n",
    "!aws s3 cp validation.manifest s3://$BUCKET/$pfx_training/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to kick off the training. We will do it from the SageMaker console, but alternatively, you can just run this code in a new cell using SageMaker Python SDK:\n",
    "### Code option\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import sagemaker\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "training_image = sagemaker.amazon.amazon_estimator.get_image_uri(\n",
    "    boto3.Session().region_name, 'object-detection', repo_version='latest')\n",
    "s3_output_path = 's3://{}/{}/output'.format(BUCKET, pfx_training)\n",
    "\n",
    "# Create unique job name\n",
    "training_job_name = 'bees-detection-resnet'\n",
    "\n",
    "training_params = \\\n",
    "    {\n",
    "        \"AlgorithmSpecification\": {\n",
    "            # NB. This is one of the named constants defined in the first cell.\n",
    "            \"TrainingImage\": training_image,\n",
    "            \"TrainingInputMode\": \"Pipe\"\n",
    "        },\n",
    "        \"RoleArn\": role,\n",
    "        \"OutputDataConfig\": {\n",
    "            \"S3OutputPath\": s3_output_path\n",
    "        },\n",
    "        \"ResourceConfig\": {\n",
    "            \"InstanceCount\": 1,\n",
    "            \"InstanceType\": \"ml.p2.xlarge\",\n",
    "            \"VolumeSizeInGB\": 50\n",
    "        },\n",
    "        \"TrainingJobName\": training_job_name,\n",
    "        \"HyperParameters\": {  # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.\n",
    "            \"base_network\": \"resnet-50\",\n",
    "            \"use_pretrained_model\": \"1\",\n",
    "            \"num_classes\": \"1\",\n",
    "            \"mini_batch_size\": \"1\",\n",
    "            \"epochs\": \"100\",\n",
    "            \"learning_rate\": \"0.001\",\n",
    "            \"lr_scheduler_step\": \"\",\n",
    "            \"lr_scheduler_factor\": \"0.1\",\n",
    "            \"optimizer\": \"sgd\",\n",
    "            \"momentum\": \"0.9\",\n",
    "            \"weight_decay\": \"0.0005\",\n",
    "            \"overlap_threshold\": \"0.5\",\n",
    "            \"nms_threshold\": \"0.45\",\n",
    "            \"image_shape\": \"300\",\n",
    "            \"label_width\": \"350\",\n",
    "            \"num_training_samples\": str(num_training_samples)\n",
    "        },\n",
    "        \"StoppingCondition\": {\n",
    "            \"MaxRuntimeInSeconds\": 86400\n",
    "        },\n",
    "        \"InputDataConfig\": [\n",
    "            {\n",
    "                \"ChannelName\": \"train\",\n",
    "                \"DataSource\": {\n",
    "                    \"S3DataSource\": {\n",
    "                        \"S3DataType\": \"AugmentedManifestFile\",  # NB. Augmented Manifest\n",
    "                        \"S3Uri\": s3_train_data_path,\n",
    "                        \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                        # NB. This must correspond to the JSON field names in your augmented manifest.\n",
    "                        \"AttributeNames\": ['source-ref', 'bees-500']\n",
    "                    }\n",
    "                },\n",
    "                \"ContentType\": \"application/x-recordio\",\n",
    "                \"RecordWrapperType\": \"RecordIO\",\n",
    "                \"CompressionType\": \"None\"\n",
    "            },\n",
    "            {\n",
    "                \"ChannelName\": \"validation\",\n",
    "                \"DataSource\": {\n",
    "                    \"S3DataSource\": {\n",
    "                        \"S3DataType\": \"AugmentedManifestFile\",  # NB. Augmented Manifest\n",
    "                        \"S3Uri\": s3_validation_data_path,\n",
    "                        \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                        # NB. This must correspond to the JSON field names in your augmented manifest.\n",
    "                        \"AttributeNames\": ['source-ref', 'bees-500']\n",
    "                    }\n",
    "                },\n",
    "                \"ContentType\": \"application/x-recordio\",\n",
    "                \"RecordWrapperType\": \"RecordIO\",\n",
    "                \"CompressionType\": \"None\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "\n",
    "# Now we create the SageMaker training job.\n",
    "client = boto3.client(service_name='sagemaker')\n",
    "client.create_training_job(**training_params)\n",
    "\n",
    "# Confirm that the training job has started\n",
    "status = client.describe_training_job(TrainingJobName=training_job_name)['TrainingJobStatus']\n",
    "print('Training job current status: {}'.format(status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check the progess of the training job, you can refresh the console or repeatedly evaluate the following cell. When the training job status reads `'Completed'`, move on to the next part of the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### REPLACE WITH YOUR OWN TRAINING JOB NAME\n",
    "# In the above console screenshots the job name was 'bees-detection-resnet'.\n",
    "# But if you used Python to kick off the training job,\n",
    "# then 'training_job_name' is already set, so you can comment out the line below.\n",
    "training_job_name = 'bees-training'\n",
    "##### REPLACE WITH YOUR OWN TRAINING JOB NAME\n",
    "\n",
    "training_info = client.describe_training_job(TrainingJobName=training_job_name)\n",
    "\n",
    "print(\"Training job status: \", training_info['TrainingJobStatus'])\n",
    "print(\"Secondary status: \", training_info['SecondaryStatus'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='review_training'></a>\n",
    "\n",
    "## Review of Training Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's create the SageMaker model out of model artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "model_name = training_job_name + '-model' + timestamp\n",
    "\n",
    "training_image = training_info['AlgorithmSpecification']['TrainingImage']\n",
    "model_data = training_info['ModelArtifacts']['S3ModelArtifacts']\n",
    "\n",
    "primary_container = {\n",
    "    'Image': training_image,\n",
    "    'ModelDataUrl': model_data,\n",
    "}\n",
    "\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "create_model_response = client.create_model(\n",
    "    ModelName = model_name,\n",
    "    ExecutionRoleArn = role,\n",
    "    PrimaryContainer = primary_container)\n",
    "\n",
    "print(create_model_response['ModelArn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "endpoint_config_name = training_job_name + '-epc' + timestamp\n",
    "endpoint_config_response = client.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType':'ml.t2.medium',\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName':model_name,\n",
    "        'VariantName':'AllTraffic'}])\n",
    "\n",
    "print('Endpoint configuration name: {}'.format(endpoint_config_name))\n",
    "print('Endpoint configuration arn:  {}'.format(endpoint_config_response['EndpointConfigArn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Endpoint\n",
    "\n",
    "The next cell creates an endpoint that can be validated and incorporated into production applications. This takes about 10 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "endpoint_name = training_job_name + '-ep' + timestamp\n",
    "print('Endpoint name: {}'.format(endpoint_name))\n",
    "\n",
    "endpoint_params = {\n",
    "    'EndpointName': endpoint_name,\n",
    "    'EndpointConfigName': endpoint_config_name,\n",
    "}\n",
    "endpoint_response = client.create_endpoint(**endpoint_params)\n",
    "print('EndpointArn = {}'.format(endpoint_response['EndpointArn']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name=\"test-tuning-job-008-9ff8af52-ep-2019-07-19-12-25-46\"\n",
    "# get the status of the endpoint\n",
    "response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = response['EndpointStatus']\n",
    "print('EndpointStatus = {}'.format(status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will invoke the deployed endpoint to detect bees in the 10 test images that were inside the `test` folder in `dataset.zip` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "test_images = glob.glob('test/*')\n",
    "print(*test_images, sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, define a function that converts the prediction array returned by our endpoint to the bounding box structure expected by our image display function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prediction_to_bbox_data(image_path, prediction):\n",
    "    class_id, confidence, xmin, ymin, xmax, ymax = prediction\n",
    "    width, height = Image.open(image_path).size\n",
    "    bbox_data = {'class_id': class_id,\n",
    "               'height': (ymax-ymin)*height,\n",
    "               'width': (xmax-xmin)*width,\n",
    "               'left': xmin*width,\n",
    "               'top': ymin*height}\n",
    "    return bbox_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, for each of the test images, the following cell transforms the image into the appropriate format for realtime prediction, repeatedly calls the endpoint, receives back the prediction, and displays the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "runtime_client = boto3.client('sagemaker-runtime')\n",
    "\n",
    "# Call SageMaker endpoint to obtain predictions\n",
    "def get_predictions_for_img(runtime_client, endpoint_name, img_path):\n",
    "    with open(img_path, 'rb') as f:\n",
    "        payload = f.read()\n",
    "        payload = bytearray(payload)\n",
    "\n",
    "    response = runtime_client.invoke_endpoint(EndpointName=endpoint_name, \n",
    "                                       ContentType='application/x-image', \n",
    "                                       Body=payload)\n",
    "\n",
    "    result = response['Body'].read()\n",
    "    result = json.loads(result)\n",
    "    return result\n",
    "\n",
    "\n",
    "# wait until the status has changed\n",
    "client.get_waiter('endpoint_in_service').wait(EndpointName=endpoint_name)\n",
    "endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = endpoint_response['EndpointStatus']\n",
    "if status != 'InService':\n",
    "    raise Exception('Endpoint creation failed.')\n",
    "\n",
    "for test_image in test_images:\n",
    "    result = get_predictions_for_img(runtime_client, endpoint_name, test_image)\n",
    "    confidence_threshold = .2\n",
    "    best_n = 3\n",
    "    # display the best n predictions with confidence > confidence_threshold\n",
    "    predictions = [prediction for prediction in result['prediction'] if prediction[1] > confidence_threshold]\n",
    "    predictions.sort(reverse=True, key = lambda x: x[1])\n",
    "    bboxes = [prediction_to_bbox_data(test_image, prediction) for prediction in predictions[:best_n]]\n",
    "    show_annotated_image(test_image, bboxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='model_tuning'></a>\n",
    "## Model Tuning\n",
    "\n",
    "When you configured the training job you needed to add many hyperparameters that affect the performance of the algorithm and the quality of the resulting model. But how do you pick the right hyperparameters?\n",
    "\n",
    "If you have experience with the specific algorithm and understand the innerworkings of it, you may already have a good sense of appropriate values. But even then, it's impossible to know the exact best value of each hyperparameter. Often you can zero in on the best values by trying many different combination of values, effectively searching in the hyperparameter space. SageMaker makes this extremely easy with the Model Tuning feature, also known as Hyperparameter Optimization (or HPO). With Model Tuning you simply decide which of the hyperparameters you are not sure about and specify the range of values for each that SageMaker needs to explore. Let's see again how this can be accomplished via the console.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='cleanup'></a>\n",
    "## Cleanup\n",
    "\n",
    "At the end of the lab we would like to delete the real-time endpoint, as keeping a real-time endpoint around while being idle is costly and wasteful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.delete_endpoint(EndpointName=endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}